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ABSTRACT 


Accurate  weather  forecasts  are  vital  to  air  combat  operations. 
Quantitative  assessments  of  forecasts  and  their  operational  impacts  are  essential 
to  improving  weather  support  for  war  fighters.  We  adapted  an  existing  U.S. 
Navy,  web-based,  near-real  time  system  for  collecting  and  analyzing  data  on  the 
performance  and  operational  impacts  of  military  forecasts.  We  used  the  adapted 
system  to  collect  and  analyze  data  on  Air  Force  Weather  (AFW)  forecasts,  and 
the  planning  and  execution  of  flying  operations,  at  six  Air  Combat  Command  and 
Pacific  Air  Forces  bases.  We  analyzed  the  data  to  develop  quantitative  metrics 
of  forecast  performance  and  operational  impacts.  Our  results  indicate  that 
planning  weather  forecasts  (PWFs)  have  a  higher  potential  for  making  positive 
contributions  to  air  operations  than  do  mission  execution  forecasts  (MEFs).  This 
is  notable  because  AFW  units  spend  significantly  less  time  developing  PWFs 
than  MEFs.  Surface  visibility,  cloud  ceilings,  and  cloud  layers  caused  most 
negative  mission  impacts,  indicating  these  phenomena  should  be  a  focus  of 
future  research  and  training.  We  found  high  levels  of  mission  success  even 
when  forecasts  were  inaccurate,  perhaps  due  to  aircrew  and  mission  flexibility. 
Our  analyses  revealed  a  need  for  improved  education  of  flying  units  on  the 
nature  and  availability  of  AFW  products. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Accurate  weather  forecasts  are  vital  to  planning  and  conducting  effective 
combat  operations.  Focused  quantitative  analyses  of  military  weather  forecasts 
and  their  operational  impacts  are  essential  to  ensure  high  quality  weather 
support  for  war  fighters.  In  today’s  resource  limited  and  changing  environment, 
more  effective  and  efficient  processes  and  procedures  are  essential.  Improved 
processes,  when  employed  throughout  an  organization,  can  save  manpower  and 
resources  and  enhance  mission  accomplishment.  In  this  study,  we  have 
analyzed  some  of  the  challenges  facing  Air  Force  weather  (AFW)  personnel, 
especially  the  difficulties  associated  with  obtaining  operator  feedback  on  mission 
execution  forecasts  (MEFs)  that  is  required  of  CWTs  (AFMAN  15-129).  To 
address  these  difficulties,  we  have  developed  and  tested  a  system  to  improve  the 
feedback  process.  The  system  was  designed  to  improve  operator  feedback  to 
the  CWTs  without  increasing  CWT  workloads. 

1 .  War  on  T errorism 

The  United  States  military  has  seen  many  changes  in  the  way  business  is 
done  since  September  11,  2001,  when  the  United  States  experienced  horrific 
attacks  by  terrorists.  In  the  Air  Force  Chief  of  Staffs  remarks  at  the  2005  Air 
Force  Defense  Strategy  and  Transformation  Seminar  Series  in  Washington  D.C. 
on  9  February  2005,  General  Moseley  indicated  there  were  about  30,000  active, 
Guard,  and  Reserve  airmen  deployed  (Moseley  2005).  In  an  Air  Force  Policy 
Letter  Digest  article  in  December  2005,  Air  Force  Secretary  Michael  W.  Wynne 
said  the  Air  Force  is  evolving  in  response  to  a  new  global  war  on  terrorism 
(Wynne  2005).  Secretary  Wynne  indicated  in  this  article  that  the  Air  Force 
deployed  more  than  300  aircraft  and  24,000  airmen  in  support  of  the  war  on 
terrorism. 

Deploying  so  many  airmen  inevitably  results  in  fewer  being  available  to 
sustain  home  station  operations.  In  many  situations,  this  causes  home  stations 
to  refocus  their  operations,  which  results  in  many  processes  and  procedures 
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being  altered.  The  new  processes  and  procedures  will  logically  be  in  the 
forefront  of  a  unit’s  focus,  while  previously  developed  tasks  (e.g.,  required  routine 
training,  administrative  and  planning  requirements)  may  be  given  a  lower  priority. 
In  the  AFW  career  field,  some  of  the  less  critical  and  less  easily  executed  tasks 
include  the  determination  of  forecast  performance  metrics  and  collection  of 
customer  feedback.  These  tasks  generally  do  not  have  firm  timelines,  and  many 
AFW  units  lack  the  expertise  to  perform  them  well.  However,  in  a  working 
environment  in  which  units  are  faced  with  increasing  deployments  and  limited 
resources,  these  tasks  can  make  AFW  units  more  effective  and  efficient.  Thus 
these  tasks  become  even  more  essential  for  achieving  overall  mission  success. 
This  highlights  the  need  for  highly  effective  and  efficient  processes  that,  once 
implemented,  can  improve  effectiveness  while  reducing  needed  resources. 

2.  Reengineering  Air  Force  Weather 

a.  Initiation  and  Implementation 

In  addition  to  deployments,  AFW  personnel  have  seen  many 
internal  changes  in  the  way  business  is  done  over  the  last  ten  years  intended  to 
increase  effectiveness  while  improving  war  fighter  support.  In  1997,  the  Air 
Force  director  of  weather  (AF/A30-W)  implemented  a  reengineering  plan  to 
realign  its  force  structure  at  the  operational  and  tactical  levels  of  warfare.  The 
changes  were  necessary  because  manpower  cuts,  among  other  factors,  caused 
AFW  to  fall  below  the  critical  personnel  levels  necessary  to  sustain  itself 
(AFWSPV  2004). 

AFW  accomplished  both  a  support  and  a  personnel  restructuring. 
Part  of  the  restructure  was  to  stand  up  operational  weather  squadrons  (OWSs) 
focused  at  the  operational  level  of  warfare.  The  OWS  took  over  production  of 
several  tasks  from  the  base  weather  station,  such  as  terminal  aerodrome 
forecasts,  weather  watches,  warnings,  and  advisories.  This  freed  up  the  combat 
weather  teams  (CWTs)  at  military  bases  to  focus  on  mission-tailored  forecasting 
for  their  supported  units  (AFWSPV  2004).  AFW  leadership  envisioned  that  by 
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focusing  the  base-level  weather  team  on  products,  such  as  MEFs,  weather 
teams  would  be  more  likely  to  become  fully  integrated  into  flying  operations 
(AFAA  2005). 

While  some  units  are  still  trying  to  fully  implement  portions  of  the 
restructuring,  many  have  been  hindered  in  doing  so  by  additional  taskings  and 
deployments  that  have  been  levied  as  a  result  of  the  war  on  terrorism.  Tasks 
such  as  daily  building  patrols,  terrorism  training,  100%  identification  checks  into 
many  base  facilities  that  had  no  checks  before,  evacuation  drills,  accountability 
drills,  and  simple  medical  training  have  inundated  an  already  full  schedule.  This 
has  resulted  in  many  personnel  and  units  becoming  task  saturated. 

As  noted  in  a  recent  Air  Force  audit  (AFAA  2005),  the  scramble  to 
keep  up  with  demands  placed  on  CWTs  by  Air  Force  instructions  and  directives 
is  challenging.  Due  to  these  challenges,  restructuring  has,  in  some  cases,  not 
achieved  its  full  potential  (AFAA  2005). 

b.  Reengineering  Challenges 

Although  the  AF/A30-W  reengineering  plan  was  implemented  with 
extensive  guidance,  some  CWTs  are  not  able  to  fully  implement  some  aspects  of 
the  vision.  After  holding  positions  out  of  the  AFW  career  field  for  six  years,  I  re¬ 
entered  it  in  June  2001  by  taking  a  position  as  the  CWT  commander  at  an  Air 
Force  installation  supporting  fighter  aircraft.  I  had  previously  read  the  Air  Force 
Instructions  (AFIs)  regarding  reengineering  and  looked  forward  to  stepping  into  a 
new  organizational  framework.  What  soon  became  apparent  was  that  the 
concept  was  challenging  to  implement,  both  from  a  CWT  and  customer 
perspective,  due  to  limited  resources,  and  perhaps  resistance  to  change  by  some 
personnel. 

It  took  me  nearly  two  years  as  a  CWT  flight  commander  to  integrate 
weather  personnel  into  two  of  the  flying  squadrons  at  my  installation.  In  fact,  one 
squadron  never  did  allow  this  during  my  time  as  the  CWT  flight  commander. 
Integration  was  only  possible  when  squadron  and  group  leadership  were  willing 
to  endure  some  of  the  pain  involved.  Often,  when  customer  leadership  changes, 
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so  does  the  integration  of  the  CWT  and  its  weather  support.  The  challenge  is  to 
institutionalize  a  process  so  that  when  leadership  changes,  weather  support  does 
not.  Once  weather  personnel  are  integrated  into  the  flying  squadrons,  it  is  much 
easier  to  obtain  operator  feedback.  Integration  helps  customers  get  to  know 
CWT  personnel,  which  in  turn  helps  the  CWT  get  customer  feedback.  The 
increase  in  feedback  our  weather  team  gained  by  integrating  our  personnel  into 
the  customers’  environment  was  invaluable.  This  feedback  enabled  us  to  make 
some  drastic  operator-driven  changes  to  our  MEFs,  which  led  to  improved 
weather  support  to  the  customer. 

Since  2003,  the  same  CWT  I  led  has  reverted  back  to  little  or  no 
integration  of  weather  personnel  into  the  flying  squadrons.  This  is,  in  part,  due  to 
decreased  manning  because  of  deployments  supporting  the  war  on  terrorism. 
Because  this  integration  concept  might  not  be  possible  at  many  CWTs,  our  study 
developed  and  tested  a  system  for  gaining  operator  feedback  for  individual 
missions  that  can  be  easily  adapted  to  almost  any  CWT.  This  system  was 
designed  to  help  streamline  CWT  processes  and  procedures. 

3.  Air  Force  Manual  15-129,  Air  and  Space  Weather  Operations  - 

Processes  and  Procedures 

CWT  operations  are  outlined  in  chapter  4  of  Air  Force  Manual  15-129,  Air 
and  Space  Weather  Operations  -  Processes  and  Procedures,  2004  (hereafter 
referred  to  as  AFMAN  15-129).  This  manual  applies  to  all  Air  Force  personnel 
and  organizations  conducting  weather  operations.  Because  each  CWT  has  its 
own  unique  mission  and  responsibilities,  the  manual  gives  general  guidance  on 
what  needs  to  be  accomplished.  Each  CWT  must  tailor  its  weather  products  to 
its  customers  in  such  a  manner  that  it  provides  environmental  information  ready 
for  use  by  customers  in  making  mission  planning  and  execution  decisions.  This 
support  can  be  provided  in  a  number  of  ways,  but  should  follow  what  AFMAN  15- 
129  calls  the  MEF  process  (MEFP).  The  purpose  of  the  MEFP  is  to  “temporally 
and  spatially  refine  forecast  products  to  provide  decision-quality  environmental 
information  for  an  operational  decision-making  process”  (AFMAN  15-129). 
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The  MEFP  is  a  twelve  step  process: 

1 )  Determine  the  what,  when,  where,  who,  and  how  of  the  primary  and 
secondary  missions. 

2)  Define  weather  thresholds. 

3)  Define  products,  services,  and  data  types. 

4)  Coordinate  operations. 

5)  Obtain  weather  situational  awareness. 

6)  Conduct  mission-scale  analysis. 

7)  Predict  mission  execution  weather  parameters. 

8)  Tailor  forecast  parameters  to  mission. 

9)  Disseminate  MEF. 

10)  Conduct  mission  watch. 

1 1 )  Update  forecast  products/information. 

12)  Conduct  MEF  verification. 

These  12  steps  fall  into  four  phases: 

1)  planning  and  coordinating  (determined  by  CWT  leadership;  steps  1- 

4), 

2)  preparing  (steps  5-8) 

3)  executing  (steps  9-10) 

4)  follow-up  (steps  11-12) 

The  MEFP  is  a  process  for  continuous  assessment  and  improvement  of 
weather  support.  To  achieve  this  improvement,  the  full  process  should  be 
routinely  used  and  its  effects  regularly  assessed.  This  is  especially  true  when 
customer  missions  change,  because  then  the  MEFs,  and  the  steps  for  evaluating 
the  MEFs,  must  also  change.  In  this  study,  we  have  focused  on  the  follow-up 
phase  of  the  MEFP  for  aviation  support.  This  is  sometimes  regarded  as  the  final 
phase,  but  is  actually  the  restart  of  the  first  phase,  and  provides  weather 
personnel  with  a  foundation  on  which  to  build  and  remodel  the  MEF. 
a.  Update  Forecast  Products/Information 
The  main  goal  of  step  eleven  is  to  improve  weather  support  by 
modifying  the  MEFs  based  on  customer  feedback.  Customer  feedback  may 
change  as  customer  leadership  changes.  But  the  resulting  changes  to  the  MEFs 
should  be  relatively  small  because  each  CWT  should  have  a  weather  support 
plan  in  place  that  outlines  the  what,  when,  and  how  of  the  CWT’s  products  and 
processes.  These  plans  help  stabilize  the  CWT’s  products  and  processes, 
ensure  well  defined  customer  support,  and  make  CWT  operations  more  efficient. 
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b.  Conduct  Mission  Verification 

Step  twelve  of  the  MEFP  directs  CWTs  to  perform  seven  tasks 
(AFMAN  15-129): 

1)  Implement  systematic  procedures  to  analyze  and  measure 
accuracy/relevancy  of  environmental  services  provided  to 
parent/host  unit. 

2)  Debrief  operator.  Face-to-face  feedback  is  preferred; 

employ  other  feedback  methods  whenever  direct  feedback 
cannot  be  obtained. 

3)  Disseminate  output  from  weather  debriefs  or  pilot  reports 
(PIREPS)  to  OWSs  and  other  weather  team  members. 

4)  Perform  technical  verification  (evaluate  forecast  skills, 
under/over  forecast,  bias,  etc.). 

5)  Conduct  operations  verification  based  on  established 
“Go/No  Go”  thresholds. 

6)  Develop  and  apply  metrics  to  process  improvement.  Use 
feedback  to  develop  rules  of  thumb  and  lessons  learned. 

7)  Accomplish,  document,  train  with,  and  cross  feed  forecast 
reviews. 

The  seven  tasks  for  step  12  are  all  important,  but,  in  practice,  they  can  be 
very  difficult  to  complete,  especially  given  the  resource  limitations  described 
earlier  in  this  chapter.  In  this  study,  we  focused  on  developing  and  testing  a 
system  to  facilitate  the  completion  of  four  of  these  tasks:  1,  2,  5,  and  6.  Our 
goals  were  to  create  a  system  that  simplified  the  process  of:  (1)  collecting  data 
with  which  to  verify  forecasts  and  assess  customer  performance;  (2)  collecting 
customer  feedback  data;  (3)  analyzing  forecast  and  customer  performance;  (4) 
relating  forecast  and  customer  performance  to  determine  the  operational  impacts 
of  the  forecasts;  and  (5)  identifying  methods  for  improving  weather  support  for 
customer  operations.  These  five  goals  directly  address  tasks  1 , 2,  5,  and  6. 

Implicit  in  these  tasks  and  goals  is  the  understanding  that  good  forecast 
performance  (e.g.,  high  forecast  accuracy)  does  not  necessarily  indicate  good 
customer  performance  (e.g.,  success  in  completing  missions).  There  are  many 
ways  in  which  accurate  forecasts  can  fail  to  provide  optimal  support  for  mission 
planning  and  execution  (e.g.,  by  failing  to  provide  relevant  forecasts  or  to  provide 
forecast  information  in  a  timely,  understandable  and  readily  useable  format) 
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The  preferred  method  to  obtain  customer  feedback  is  face-to-face 
discussion  (AFMAN  15-129).  However,  this  is  not  always  possible  due  to 
resource  constraints.  Further,  if  a  CWT  has  educated  its  customers  on  weather 
impacts  and  the  customers  know  what  feedback  is  needed,  then  a  face-to-face 
debrief  may  not  be  needed.  Whatever  the  method  for  collecting  customer 
feedback,  it  is  important  that  the  feedback  comes  from  the  customer  and  not  from 
CWT  personnel  inferring  what  the  customers  would  say  if  asked.  The  system  we 
developed  for  this  study  was  designed  to  collect  data  directly  from  aircrews  on 
the  accuracy  of  the  forecasts  they  received,  the  success  of  their  missions,  the 
usefulness  and  specific  impacts  of  the  forecasts  on  the  planning  and  execution  of 
their  missions,  the  weather  phenomena  that  were  encountered  during  the 
mission,  and  the  impacts  of  those  phenomena  on  the  execution  of  the  mission.. 

4.  Air  Force  instruction  15-114,  Functional  Resource  and  Weather 
Technical  Performance  Evaluation 

Guidance  and  procedures  for  measuring  and  evaluating  the  operational 
effectiveness  and  technical  performance  of  atmospheric  and  space 
environmental  support,  collectively  termed  weather  support,  is  defined  in  Air 
Force  Instruction  15-114,  Functional  Resource  and  Weather  Technical 
Performance  Evaluation,  2001  (hereafter  referred  to  as  AFI  15-114).  This  14- 
page  document  defines  three  categories  of  forecast  assessment: 

1)  forecast  impacts  on  mission  execution  determined  by  operational 
verification  (OPVER) 

2)  forecast  accuracy  determined  by  aerodrome  forecast  verification 

3)  resource  protection  effectiveness  determined  by  warning/advisory 
verification  (WARNVER) 

a.  OPVER 

AFI  15-1 14  calls  operational  verification  of  the  MEF  (MEFVER)  the  “single 
most  important  mission-oriented,  operational  effectiveness  assessment 
requirement  for  CWTs.”  The  instruction  further  explains  that  this  type  of  OPVER 
will  be  used  by  the  Air  Staff  to  explain  AFW  capabilities  to  Department  of 
Defense  and  National  senior  leaders.  According  to  AFI  15-114,  “CWTs  develop 
OPVER  criteria  through  close  coordination  with  operators.  CWTs  then  collect 
verification  data  by  debriefing  customers  and/or  analyzing  observed  versus 
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forecast  conditions  to  determine  forecast  impact  on  tactical-level  mission 
effectiveness.”  However,  most  CWTs  do  not  have  the  resources  or  expertise  to 
develop  such  an  OPVER  process.  Our  study  is  intended  to  help  correct  this 
problem  by  streamlining  the  OPVER  process. 

b.  Automating  Readiness  and  Technical  Performance 
Metrics 

Paragraph  two  of  AFI  15-1 14  calls  for  a  system  that,  “automates  all 
metrics,  from  data  collection,  aggregation,  to  data  quality  control.  The  end-state 
will  be  an  automated  web-based  system  with  the  capability  to  provide  ad  hoc 
analyses  and  reports  (assessments)  for  all  levels  of  AFW  support.”  The 
instruction  also  provides  guidance  on  how  to  proceed  until  this  automation  is 
realized.  The  system  we  have  developed  in  this  study  automates  many  of  the 
steps  needed  to  calculate  the  required  metrics. 

5.  Air  Force  Agency  Report 

The  Air  Force  Audit  Agency  (AFAA)  issued  a  “Weather  Operations 
Metrics”  report  on  27  September  2005  (AFAA  2005),  in  response  to  a  request 
from  AF/A30-W,  to  determine  whether  or  not  the  Air  Force  effectively 
implemented  MEFVER.  Specifically,  the  AFAA  was  asked  to  determine  if 
weather  personnel  effectively: 

1)  accomplished  mission  execution  forecast  verification 

2)  included  mission  operator  feedback  on  weather  forecasts  in 

MEFVERs 

3)  identified  improvement  opportunities  (AFAA  2005) 

There  were  19  CWTs  reviewed  in  the  audit,  three  of  which  also 

participated  in  our  research  project.  The  audit  was  performed  during  February- 

April  2005  using  documents  dated  13  March  1998  through  17  March  2005.  The 

audit  found  that  AFIs  do  not  provide  standardized  procedures  for  obtaining 

mission  operator  feedback  on  weather  forecasts  and  do  not  require  operations 

personnel  to  provide  formal  feedback  on  weather  forecast  accuracy  and  mission 

impact.  The  audit  found  that  low  customer  feedback  rates  had  a  negative  impact 

on  weather  teams’  efforts  to  identify  improvement  opportunities.  Specifically,  the 

audit  mentioned  that  low  response  rates  from  customers  hindered  data  usage 

and  prevented  weather  teams  from  deriving  any  usable  information  from 
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customer  feedback.  The  audit  also  indicated  that  the  relevant  AFIs  did  not 
provide  sufficient  guidance  or  clear  procedures  for  identifying  improvement 
opportunities  within  the  MEFVER  program  (AFAA  2005). 

The  audit  team  made  several  recommendations  to  the  Deputy  Chief  of 
Staff,  Air  and  Space  Operations  (AF/XO).  The  recommendations  that  apply  to 
this  study  are  to: 

1)  establish  standardized  procedures  for  obtaining  mission  operator 
feedback  on  weather  forecasts 

2)  direct  AF/XOO-W  and  Air  Force  Director  of  Operational  Training 
(AF/XOO-T)  to  coordinate  and  amend  existing  processes  to  require 
operators  to  provide  formal  feedback  on  weather  forecast  accuracy 
and  mission  impact. 

The  AF/A3  concurred  with  the  audit  results  and  stated  they  would  be 
rewriting  and  updating  AFI  15-114  to  establish  standardized  procedures  for 
obtaining  mission  operator  feedback  on  weather  forecasts  and  identifying 
improvement  opportunities  within  the  MEFVER  program,  with  an  estimated 
completion  date  of  1  March  2006.  The  AF/A3  also  agreed  to  amend  existing 
processes  to  require  operators  to  provide  formal  feedback  on  weather  forecast 
accuracy  and  mission  impact  with  an  estimated  completion  date  of  1  November 
2005. 

Some  of  the  problems  cited  in  the  audit  report  could  be  fixed  if  CWTs  were 
fully  integrated  into  the  flying  squadrons  they  support.  AFW  does  not  have  the 
manpower  to  this,  so  other  fixes  must  be  found.  In  our  study  we  developed  a 
potential  solution  to  the  problem  of  Air  Force  personnel  not  consistently  and 
completely  obtaining  customer  feedback  and  identifying  improvement 
opportunities.  The  audit  recommended  AFW  establish  standardized  procedures 
for  obtaining  mission  operator  feedback  on  weather  forecasts.  Further,  the  audit 
directed  Air  Staff  agencies  to  coordinate  and  amend  existing  processes  to 
require  operators  to  provide  formal  feedback  on  weather  forecast  accuracy  and 
mission  impact  (AFAA  2005).  Our  research  project  provides  a  process  for 
meeting  these  recommendations  and  directives. 
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6.  Air  Force  Weather  Strategic  Plan  and  Vision 

The  AFW  Strategic  Plan  and  Vision  (AFWSPV)  issued  August  2004 
(AFWSPV  2004)  was  intended  to  set  AFW’s  course  for  transformation  starting 
with  fiscal  year  2008.  The  plan  outlines  eight  “transformation  vectors”  that  guide 
the  weather  core  processes  of  collection,  analysis,  prediction,  tailoring,  and 
dissemination.  The  eight  vectors  are: 

1)  Integrate  environmental  information  collection  into  the  theater 
intelligence,  surveillance,  and  reconnaissance  collection  plan. 

2)  Anticipate  and  manage  the  data  explosion  from  the  next  generation 
environmental  satellites,  unmanned  vehicles,  national  technical 
means,  smart  tankers,  future  combat  system  vehicles,  etc. 

3)  Advocate  space  weather  models  going  from  climatologically  and 
statistically  based  to  physics  based. 

4)  Anticipate  and  manage  increasing  model  resolution,  vertical  domain 
from  surface  to  near  space,  and  physics  requirements  based  on 
new  weapon  systems  coming  into  the  inventory. 

5)  Move  from  a  “graphically  based”  to  “digitally  based”  product  line. 

6)  Exploit  automated  decision  tools  as  a  mainstream  product  (some 
will  be  machine-to-machine  (M2M),  others  will  be  semi-automated). 

7)  Fully  integrate  weather  information  into  decision  cycles  at  the 
appropriate  levels  of  command. 

8)  Fully  embed  people  with  users  as  “weather/climate  consultants” 

In  our  study,  we  have  addressed  the  last  three  vectors,  especially  in  terms 
of  exploiting  automation  (vector  6),  determining  where  weather  information  is 
integrated  into  aviator  decision  cycles,  from  the  base  level  down  (vector  7),  and 
the  value  of  embedded  weather  personnel  in  flying  squadrons  (vector  8). 

AFWSPV  (2004)  states,  “if  you  ‘boiled  down’  the  AFW  business  to  its 
essence,  you  would  find  that  we  integrate  into  operations  and  intelligence,  that 
our  analyses  provide  battlefield  situational  awareness,  and  our  predictions  and 
tailored  products  enhance  decision  superiority  for  commanders  at  every  level” 
(AFWSPV  2004).  Our  study  provides  a  prototype  system  for  enabling  improved: 
(1)  integration  of  AFW  personnel  into  operations;  (2)  CWT  awareness  of  how 
aviators  view  and  use  AFW  products  to  plan  and  execute  their  missions;  and  (3) 
tailoring  of  products  to  improve  customer  decision  making. 
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B.  PREVIOUS  WORK 

1.  Metrics  of  Forecast  Performance  and  Impacts  on  War  Fighting 
Operations 

In  2005,  Captain  Jeff  Jarry,  USAF,  (Jarry  2005)  performed  analyses  of 
MEFVER  data  for  FY  2004  provided  by  the  Air  Mobility  Command  (AMC).  The 
three  main  motives  behind  his  study  were  to: 

1)  determine  if  AFW  personnel  within  AMC  were  adding  value  to  war 
fighting  operations 

2)  improve  the  collection  and  analysis  of  AFW  MEFVER  data 

3)  provide  constructive  inputs  for  the  review  and  update  process  for 
the  Air  Force  regulations  that  govern  metrics  verification  programs 
and  procedures 

Jarry  (2005)  analyzed  MEFVER  data  from  13  CWTs.  However,  some  of 
the  CWTs  did  not  provide  data  on  all  of  the  MEFs  they  issued  and  it  was  not 
clear  how  many  missions  were  actually  supported  by  the  reported  MEFs.  Jarry’s 
results  clearly  indicated  a  need  for  a  more  automated  and  near-real  time  system 
for  collection  and  analyses  of  data. 

Jarry  (2005)  also  found  that  “the  main  return  on  investment  in  the  CWTs 
probably  comes  from  their  forecasts  of  relatively  uncommon,  but  mission  critical, 
No  Go  conditions”.  A  No  Go  forecast  is  issued  when  one  or  more  phenomena 
are  forecasted  to  be  below  thresholds  as  determined  by  weather  personnel. 
Weather  personnel  receive  the  thresholds  from  their  customers. 

Jarry  (2005)  points  out  that  MEFVER  data  collection  needs  to  be  done  in  a 
more  consistent  and  complete  way.  Further,  he  states  that,  “(CWTs)  also  need 
to  record  for  each  MEF  how  operators  used  the  forecasts  and  the  successes  and 
failures  of  the  corresponding  mission.  If  this  data  were  collected  and 
appropriately  archived  and  analyzed,  AFW  would  be  in  a  much  better  position  to 
objectively  and  quantitatively  demonstrate  both  its  performance  and  the  value  it 
provides  to  operations,  and  to  do  so  on  a  near-real  time  basis.”  In  our  study  we 
have  developed  and  tested  a  system  for  meeting  this  need. 

2.  System  for  Conducting  Quantitative  Near  Real  Time  Analyses 

In  September  2005,  Lieutenant  Commander  Mark  Butler,  USN,  completed 

a  project  designed  to  develop  and  operationally  implement  an  online  system  for 
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collecting  and  analyzing  data  on  the  performance  and  operational  impacts  of 
forecasts,  and  then  reporting  the  results  of  the  analyses  in  near  real  time  (Butler 
2005).  The  results  of  this  project  were: 

1)  identification  of  the  data  necessary  for  quantitatively  determining 
forecast  performance  and  the  operational  impacts  of  those 
forecasts 

2)  development  of  an  online  data  collection  system  linked  to  an  online 
database  for  archiving  and  analyzing  the  required  data  into  a 
readily  accessible  form 

3)  development  of  a  web-based  interface  that  would  provide  users 
with  near  real-time  access  to  the  metrics  results  (i.e.,  the  results  of 
the  data  analyses) 

4)  development  of  a  final  metrics  report  format  that  fully  supported  the 
needs  of  the  users  (i.e.,  Naval  meteorology  and  oceanography 
(METOC)  and  AFW  units  and  their  operational  customers) 

The  system  was  tested  and  operationally  implemented  at  the  Naval  Strike 
and  Air  Warfare  Center  at  Naval  Air  Station  Fallon.  The  system  has  been 
deemed  quite  effective  by  the  METOC  officer  in  charge  of  Naval  Pacific 
Meteorology  and  Oceanography  Detachment  (hereafter  referred  to  as  NPMOD) 
at  NAS  Fallon  (Cantu  2005).  NPMOD  provides  METOC  support  to  Naval  strike 
airwings  preparing  for  upcoming  deployments.  A  major  feature  of  the  system  is 
its  ability  to  assess  the  operational  impacts  of  METOC  support,  a  process  that  at 
the  time  of  the  Butler  study  did  not  exist  in  any  other  civilian  or  military 
organization  (Butler  2005). 

C.  GOALS  OF  THIS  STUDY 

We  designed  our  study  to  address  the  directives,  requirements, 
recommendations,  needs,  and  challenges  described  in  the  preceding  sections  of 
this  chapter.  We  focused  in  particular  on  developing  methods  for  improving  the 
collection  and  analysis  of  aviator  feedback  on  the  forecasts  they  received  in 
support  of  their  missions.  Using  this  feedback,  we  calculated  metrics  of  the 
performance  and  operational  impacts  of  these  forecasts,  from  the  perspective  of 
the  aviation  customers.  We  chose  to  focus  on  weather  support  for  air  combat 
operations  in  two  major  commands,  Air  Combat  Command  (ACC)  and  Pacific  Air 
Forces  (PACAF).  Working  with  these  major  commands,  gave  us  the  opportunity 
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to  assess  weather  support  for  a  wide  range  of  platforms,  including  fighters, 
bombers,  and  unmanned  aerial  vehicles. 

Our  first  goal  was  to  adapt  the  Butler  (2005)  web-based,  near-real  time 
system  for  collecting  and  analyzing  AFW,  ACC,  and  PACAF  data  in  order  to 
assess: 

1 )  forecast  performance 

2)  weather  phenomena  with  operational  impacts 

3)  operational  impacts  of  those  forecasts  and  phenomena. 

Our  emphasis  was  on  the  third  of  these  assessments. 

Our  second  goal  was  to  implement  and  test  our  system  with  ACC  and 
PACAF  CWTs.  Our  objective  was  to  create  a  highly  streamlined  and  automated 
system  so  that  there  would  be  little  or  no  additional  work  for  the  CWTs  and  their 
customers  to  do. 

Our  third  goal  was  to  use  the  results  from  system  implementation  to 
provide  AFW,  ACC,  and  PACAF  personnel  with  advice  on  how  to  alter 
procedures  and  allocation  of  resources  to  better  accomplish  their  goals.  Our 
objectives  were  to  calculate  and  provide  forecast  performance  and  operational 
impacts  metrics,  improve  terminal  forecast  reference  notebooks,  provide  advice 
on  the  alteration  of  MEFs  and  instrument  refresher  courses,  and  provide  near- 
real  time  results  in  a  format  ready  for  use  by  leadership. 

Our  intention  was  to  provide  improved  tools  for  CWTs  to  use  in  meeting 
the  requirements  described  in  AFMAN  15-129  and  AFI  15-114,  and  for 
leadership  to  use  in  addressing  the  findings  in  the  AFAA  (2005)  report.  We  also 
set  out  in  this  study  to  make  progress  along  three  of  the  AFWSPV  vectors,  6-8  — 
in  particular  by  helping  to  automate  CWT  processes,  quantifying  the  use  and 
impacts  of  weather  information  in  customer  decision  making  processes, 
assessing  the  value  of  embedded  CWT  personnel,  and  minimizing  the  impacts 
of,  or  reducing  the  need  for,  full  integration. 
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II.  DATA  AND  METHODS 


A.  DATA  COLLECTION  METHODS 

1.  Initiation  of  Data  Collection  Efforts 

Our  data  collection  efforts  were  focused  on  fighter  aircraft  for  several 
reasons.  First,  fighters  represent  a  significant  AF  weapon  system.  Second, 
fighters  were  chosen  to  expand  on  other  studies,  such  as  Jarry  (2005)  which 
covered  AMC  air  transport  aircraft.  Finally,  the  majority  of  my  weather 
experience  has  been  in  support  of  fighter  aircraft. 

To  initiate  our  collection  efforts,  we  contacted  the  standardization  and 
evaluation  branch  in  the  weather  division  at  headquarters  ACC  to  obtain  data 
from  CWTs  supporting  ACC  assets.  ACC  is  the  primary  provider  of  combat 
airpower  to  U.S.  war  fighting  commands.  To  support  global  implementation  of 
national  security  strategy,  ACC  operates  fighter,  bomber,  reconnaissance,  battle- 
management,  and  electronic-combat  aircraft.  ACC  also  provides  command, 
control,  communications  and  intelligence  systems,  and  conducts  global 
information  operations. 

The  ACC  web  site  (ACC  2006)  explains  their  mission  as  a  force  provider, 
organizing,  training,  equipping,  and  maintaining  combat-ready  forces  for  rapid 
deployment  and  employment  to  meet  the  challenges  of  peacetime  air  sovereignty 
and  wartime  air  defense.  ACC  numbered  air  forces  provide  air  components  to 
U.S.  Central  and  Southern  Commands,  with  Headquarters  ACC  serving  as  the 
air  component  to  U.S.  Northern  and  Joint  Forces  Commands.  ACC  also 
augments  U.S.  European,  Pacific  and  Strategic  Commands  forces  (ACC  2006). 

To  support  our  project,  the  ACC  director  of  weather  contacted  all  the  ACC 
CWTs  encouraging  support  of  our  project.  Five  CWTs  agreed  to  collect  data  for 
our  project.  These  CWTs  support  a  variety  of  ACC  aircraft,  including  A-IOs,  B- 
1s,  B-2s,  F-16s,  and  U-2s.  Three  of  the  CWTs  were  involved  in  the  AFAA 
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metrics  audit  (AFAA  2005).  This  is  important  to  note  because  our  project  was 
designed  to  provide  a  system  for  assisting  CWTs  in  addressing  some  of  the 
deficiencies  noted  in  the  audit. 

After  coordination  and  initiation  of  data  collection  with  the  ACC  CWTs,  we 
expanded  our  data  to  include  two  CWTs  in  PACAF.  PACAF’s  primary  mission  is 
to  provide  ready  air  and  space  power  to  promote  U.S.  interests  in  the  Asia- 
Pacific  region  during  peacetime,  through  crisis,  and  in  war  (PACAF  2006). 
CWTs  in  PACAF  have  fully  integrated  weather  personnel  into  the  flying 
squadrons.  This  gave  us  an  opportunity  to  collect  data  that  could  be  used  to 
assess  the  value  added  by  embedding  weather  personnel.  We  focused  our 
efforts  on  collecting  data  from  two  PACAF  units  in  Korea  because  they  are  more 
operational  and  fly  more  missions  than  CONUS  units. 

2.  Know  Your  Customer 

Our  process  of  coordinating  with  the  ACC  CWTs  started  with  us  providing 
guidance  on  building  customer  relationships  and  collecting  customer  feedback. 
This  section  and  the  following  section  describe  the  major  steps  we  recommended 
be  taken  by  the  CWTs  to  improve  their  knowledge  of,  and  interaction  with,  their 
customers.  Our  objective  was  to  facilitate  the  participation,  and  increase  the 
effectiveness,  of  the  CWTs  in  the  customer  feedback  portion  of  our  project, 
a.  Understand  Supported  Mission  and  Aircraft 
The  first  step  is  developing  a  firm  understanding  of  the  supported 
mission  and  aircraft.  Without  this  understanding,  collection  of  useful  data  will  be 
difficult.  This  involves  obtaining  weather  thresholds  defined  in  the  weather 
support  plan.  After  reviewing  the  plan,  one  must  visit  the  supported  flying 
squadrons  and  obtain  weather  sensitivities  from  squadron  leadership,  planners, 
and  safety  officers.  This  information  then  needs  to  be  combined  with  weather 
threshold  data  to  develop  a  basis  for  tailored  forecasts. 

Good  references  to  use  in  the  process  of  understanding  supported 
aircraft  are  1 1-series  AFIs  covering  flying  operations.  AFI  1 1-214,  Air  Operations 
Rules  and  Procedures,  2001  (hereafter  referred  to  as  AFI  11-214),  provides  rules 

and  procedures  for  air-to-air  and  air-to-surface  operations  and  training.  The  AFIs 
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give  overarching  guidance  that  is  often  supplemented  by  MAJCOMs  and  Air 
Force  installations.  These  supplements  will  never  be  more  lenient  when  it  comes 
to  thresholds,  but  may  be  more  restrictive,  so  it  is  necessary  to  review  all 
guidance  from  the  AFIs  and  the  supplements,  from  the  MAJCOM  supplements 
right  down  to  those  from  the  base,  group,  and  squadron. 

An  excellent  location  to  find  weather  limitations  for  specific  aircraft 
is  in  a  document  called  the  Operational  Weather  Limiters  Search  (OWL).  The 
OWL  can  be  found  on  the  website  of  the  15th  Operational  Weather  Squadron 
(OWL  2006).  This  is  not  an  official  Air  Force  document  but  can  still  be  helpful  for 
general  guidance  and  is  a  good,  easy  to  read,  starting  point  for  aircraft 
thresholds.  Once  the  general  guidance  is  procured,  interviews  with 
standardization  and  evaluation  staff  in  the  operations  group  can  help  firm  up  the 
limitations  and  thresholds  for  the  specific  aircraft  supported.  In  addition,  the 
airfield  operations  staff  will  be  able  to  provide  takeoff  and  landing  criteria  for  the 
airfield. 

Finally,  weather  personnel  must  have  a  firm  understanding  of  the 
aircrew  they  support.  There  are  differing  takeoff  and  landing  criteria  based  on  an 
aircrew’s  experience  level.  It  is  important  to  know  those  criteria  so  the  thresholds 
can  be  incorporated  into  the  forecasts.  The  flying  squadron  director  of 
operations  can  be  interviewed  to  gain  knowledge  about  aircrew  in  upgrade 
training,  average  number  of  days  per  week  individuals  fly,  crew  rest 
requirements,  and  overall  demands  placed  on  flying  customers.  Integration  with 
the  supported  unit  is  important. 

b.  Develop  a  Timeline  of  How  Business  is  Conducted 
Once  the  customer  is  known,  a  basic  timeline  of  the  customer’s 
planning,  execution,  and  debriefing  process  needs  to  be  defined.  This  timeline 
should  start  at  the  point  where  decisions  are  first  made  at  the  particular 
installation.  The  wing  scheduling  office  is  a  good  place  to  start  because  this 
office  typically  coordinates  the  basic  flying  schedule  for  the  entire  wing.  Once  the 
general  flow  of  aviation  operations  is  understood,  then  a  general  weather  support 

timeline  can  be  developed.  Figure  1  shows  a  schematic  operational  timeline  with 
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decision  points  at  which  weather  products  can  be  injected  into  the  decision 
making  process.  Long  range  planning,  such  as  the  scheduling  of  exercises  and 
deployments  by  wing  offices,  will  use  climatological  weather  products,  whereas 
an  aircrew  planning  a  flight  they  will  execute  within  three  hours  will  use  a  MEF. 

Specific  customer  requirements  are  very  important  to  know  to 
determine  times  when  weather  support  is  needed,  and  what  type  of  support  is 
required  (e.g.  cloud  ceilings,  surface  visibility,  satellite  image,  thermal  crossover 
data),  and  what  format  should  be  used  (e.g.  electronic  copy,  face-to-face 
briefing).  This  information  is  critical  in  order  to  provide  the  support  the  customer 
needs  to  make  decisions  on  tactics,  weapon  type,  and  scheduling. 

3.  Design  of  Data  Collection  Process 

A  firm  understanding  of  the  customer’s  processes  is  also  critical  to  the 
collection  of  the  data  needed  to  assess  forecast  performance  and  the  operational 
impacts  of  forecasts.  For  example,  knowledge  of  the  points  within  the  customer’s 
processes  at  which  weather  information  is  used  by  the  customer  is  critical  for 
determining  what  data  needs  to  be  collected  on  the  weather  information  provided 
to  the  customer  and  the  decisions  made  by  the  customer  based  on  that 
information. 

Figure  2  shows  a  schematic  flow  chart  for  planning,  executing,  and 
debriefing  air  operations  (left  column),  and  the  corresponding  data  collection 
needed  to  analyze  the  performance  and  operational  impacts  of  the  weather 
support  provided  for  the  operations  (right  column).  The  chart  starts  at  the  wing 
level  and  works  its  way  down  to  the  mission  execution  and  debrief  of  the  mission. 
The  mission  debrief  is  crucial  because  this  is  where  weather  personnel  can  learn 
how  the  weather  products  they  provided,  and  the  weather  phenomena  actually 
encountered,  impacted  the  mission.  Information  from  debriefs  is  a  major  part  of 
the  foundation  for  improving  weather  support  to  aviators,  and  a  focal  point  in  our 
data  collection  process.  Much  of  our  study  focused  on  developing  methods  for 
efficiently  collecting  information  from  aviators  on  how  the  forecasts  and  the  actual 
weather  encountered  affected  their  missions. 
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In  order  to  determine  the  types  of  data  that  needed  to  be  collected,  we 
developed  flow  charts  depicting  all  the  weather  related  steps  in  the  mission,  from 
the  receipt  of  a  planning  weather  forecast  all  the  way  to  a  mission  debrief.  These 
flow  charts  describe  three  main  events  in  mission  planning,  execution,  and  post¬ 
mission  assessment: 

1)  Once  a  planning  weather  forecast  is  received  by  an  aviator,  a 
determination  is  made  by  the  aviator  as  to  whether  or  not  the 
mission  plan  should  be  changed. 

2)  The  aviator  then  receives  the  mission  execution  forecast  and  once 
again  determines  whether  or  not  the  mission  plan  should  be 
changed. 

3)  Unless  the  crew  decides  to  cancel  the  mission,  it  is  flown  and  the 
crew  determines  whether  or  not  weather  negatively  impacted  the 
mission  and  whether  the  mission  was  successful  or  unsuccessful. 

In  developing  our  flow  charts,  we  found  it  helpful  to  work  backward  from 
an  end  result  of  either  a  successful  mission  or  an  unsuccessful  mission.  Figure  3 
shows  all  the  ways  a  successful  mission  can  result  from:  (1)  a  planning  weather 
forecast  (PWF)  indicating  a  negative  mission  impact;  and  (2)  a  PWF  indicating  no 
negative  mission  impact.  Figure  4  shows  all  the  ways  an  unsuccessful  mission 
can  result  from  both  of  these  types  of  PWFs.  The  process  is  essentially  the 
same  as  for  successful  missions.  For  each  figure,  there  are  16  paths  shown  by 
arrows.  These  paths  represent  the  different  types  of  weather  information  that 
could  be  provided  to  an  aircrew,  the  different  types  of  decisions  the  aircrew  could 
make  in  response  to  that  information,  the  different  impacts  the  actual  weather 
could  have  on  the  mission,  and  the  different  types  of  mission  outcome. 

Our  data  collection  process  was  designed  to  collect  the  data  needed  to 
determine,  for  each  mission,  which  of  these  paths  was  followed.  Our  data 
analysis  process  was  designed  to  determine  how  and  to  what  extent  weather 
forecasts,  and  the  weather  phenomena  encountered  during  the  mission,  affected 
aviator  decisions  and  mission  success. 
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Once  we  determined  what  data  needed  to  be  collected,  we  developed  a 
generic  data  collection  form  to  be  completed  by  aircrews.  Figure  5  shows  this 
form,  which  is  broken  into  three  main  sections:  operational  impacts,  forecast 
performance,  and  target  acquisition  weapons  software  (TAWS)  data.  TAWS 
data  encompasses  both  operational  and  forecast  performance  information, 
although  not  all  aircraft  use  TAWS. 

Note  that  the  data  collected  via  this  form  is  focused  on  weather 
phenomena  that  were  expected  to  have,  and/or  did  have  negative  impacts  on  the 
mission.  This  is  because  we  chose  to  limit  the  scope  of  our  project  to  the 
phenomena  that  are  most  difficult  to  forecast  and  that  have  the  most  potential  for 
altering  mission  plans  and  hindering  mission  execution,  which  are  the  negatively 
impacting  phenomena  (Hinz  2004,  Jarry  2005,  Butler  2005). 

Note  also  that  this  form  collects  data  solely  from  aviators.  Thus,  the  data 
collected  via  this  form  represents  aviator  reports  of:  (1)  negative  weather  impacts 
the  aviators  inferred  from  the  forecasts  they  received;  (2)  the  changes  the 
aviators  made  to  their  mission  plans  in  response  to  the  negative  impacts  they 
inferred  from  the  forecasts;  (3)  the  weather  phenomena  encountered  by  the 
aviators  during  their  missions;  and  (4)  the  changes  made  by  the  aviators  to  their 
missions  due  to  the  negatively  impacting  weather  phenomena  they  encountered; 
and  (5)  the  degree  of  success  of  their  missions. 

We  had  two  main  motivations  for  collecting  this  data  solely  from  the 
aviators.  First,  we  wanted  to  develop  and  test  a  process  for  meeting  the 
requirements  imposed  on  CWTs  to  collect  and  evaluate  customer  feedback. 
Second,  the  aviators  commonly  make  their  own  inferences  from  the  forecasts 
about  negative  weather  impacts,  and  do  not  rely  on  the  negative  impacts  that 
may  or  may  not  be  provided  by  the  CWTs.  Thus,  In  effect,  the  aviators  are 
acting  as  forecasters.  They  are  at  least  acting  as  forecasters  of  the  negative 
impacts,  and  perhaps  also  of  the  weather  itself,  if  they  do  not  assume  the 
forecasts  are  accurate  (i.e. ,  if  they  do  not  believe  the  forecasts).  Thus,  we  felt  it 
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was  important  to  assess  the  performance  and  operational  impacts  of  the  aviators’ 
forecasts.  These  issues  are  discussed  further  in  the  data  analysis  section 
(section  F)  below. 

Figure  6  shows  a  separate  generic  data  collection  form  developed  to  gain 
feedback  on  a  relatively  new  program  called  infrared  target  scene  simulation 
(IRTSS).  IRTSS  software  creates  a  three-dimensional  picture  of  what  the  target 
area  will  look  like  under  specific  environmental  conditions.  IRTSS  has  not  yet 
been  fielded  throughout  the  Air  Force,  but  is  being  used  by  at  least  one  of  the 
bases  participating  in  this  study,  as  part  of  the  military’s  evolution  toward  effects- 
based  operations  (AFWSPV  2004).  The  IRTSS  data  collection  form  is  relatively 
basic  because  the  IRTSS  program  is  still  in  its  early  stages. 

B.  CWT  DATA  COLLECTION 

The  generic  data  collection  form  was  sent  to  the  participating  CWTs  with  a 
request  to  tailor  the  form  to  reflect  their  specific  customers’  missions.  When 
drafting  answer  options  for  each  question,  the  CWTs  were  directed  to  offer 
enough  options  to  cover  what  they  anticipated  to  be  80  percent  of  their 
customers’  likely  responses.  The  different  CWTs  support  very  different  missions, 
which  proved  to  be  challenging  in  developing  a  relatively  uniform  data  collection 
and  analysis  process. 

1.  Beale  AFB 

a.  Supported  Missions 

The  Beale  AFB  web  site  (Beale  2006)  provides  the  following 
description  of  the  mission  of  the  9th  Reconnaissance  Wing:  “[It]  is  responsible  for 
providing  national  and  theater  command  authorities  with  timely,  reliable,  high- 
quality,  high-altitude  reconnaissance  products.  To  accomplish  this  mission,  the 
wing  is  equipped  with  the  nation’s  fleet  of  U-2  and  RQ-4  reconnaissance  aircraft 
and  associated  support  equipment”.  There  are  also  T-38  and  reserve  KC-135 
aircraft  stationed  at  Beale  AFB,  California. 

The  U-2  provides  high-altitude,  all-weather  surveillance  and 
reconnaissance,  day  or  night,  in  direct  support  of  U.S.  and  allied  forces.  It 

delivers  critical  imagery  and  signals  intelligence  to  decision  makers  throughout  all 
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phases  of  conflict,  including  peacetime  indications  and  warnings,  low-intensity 
conflict,  and  large-scale  hostilities  (Factsheets  2006). 


The  Beale  CWT  collected  data  for  U-2  operations  because  of  the 
U-2’s  unique  missions  and  thresholds,  and  because  U-2  units  have  a  more 
formal  debrief  process  which  increased  collection  opportunities.  The  U-2 
performs  both  “high”  and  “low”  altitude  flights.  The  low  flights  receive  their  MEF 
via  a  computer  terminal  in  the  squadron,  but  the  high  flights  actually  go  to  the 
weather  station  to  receive  their  MEF.  Because  of  the  direct  interaction  with  the 
aircrews,  data  was  collected  on  the  high  flights.  The  generic  data  collection  form 
(Figure  5)  was  only  slightly  tailored  for  the  U-2  high  flights. 
b.  Data  Collection  Methods 

The  Beale  CWT  attempted  to  collect  data  in  November  and 
December  2005,  but  had  little  success  in  getting  pilots  to  return  the  forms.  The 
data  collection  forms  were  being  handed  to  the  aviators  when  they  received  their 
briefing  at  the  weather  station  with  the  request  that  they  be  filled  out  and  returned 
after  the  mission.  After  little  success  in  getting  pilots  to  fill  out  and  return  the 
forms,  the  CWT  emailed  the  forms  to  the  pilots  during  an  intensive  data  collection 
period  on  9-13  January  2006  and  was  more  successful  in  getting  completed 
forms  from  the  pilots. 

2.  Creech  AFB 

a.  Supported  Missions 

The  mission  at  Creech  AFB,  Nevada  has  evolved  significantly  in 
the  past  year  (Creech  2005).  In  March  2005,  the  Air  Force  unmanned  aerial 
vehicle  (UAF)  center  of  excellence  (COE)  was  stood  up,  and  on  20  June  2005, 
the  Indian  Springs  Air  Force  auxiliary  air  field  was  renamed  Creech  AFB  in  honor 
of  General  Wilbur  L.  “Bill”  Creech.  October  2005  saw  the  standup  of  the  Joint 
UAV  COE  at  Creech  AFB  (Creech  2005). 

Creech  AFB  is  home  to  the  11th  and  15th  Reconnaissance 
Squadrons  (Creech  2006),  whose  primary  mission  is  to  provide  theater 
commander  in  chiefs  (CINCs)  with  deployable,  long  endurance,  real-time  aerial 

reconnaissance,  surveillance,  and  target  acquisition  flying  the  RQ-1A  Predator. 
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The  Predator  reports  battlefield  conditions  to  war  fighters  using  medium-altitude, 
multi-sensor,  platforms  and  also  collects  and  distributes  imagery  products  to 
CINCs  and  national  level  leadership  (Creech  2006).  Additionally,  the  11th 
Reconnaissance  Squadron  conducts  all  Predator  aircrew  qualification  training 
(Creech  2006).  The  generic  data  collection  form  (Figure  5)  was  not  changed  for 
Creech  AFB  because  the  CWT  had  difficulty  coordinating  the  form  with  aviators. 
b.  Data  Collection  Methods 

The  weather  flight  commander  at  Creech  AFB  found  little  to  no  time 
to  coordinate  the  generic  data  collection  form  and  therefore  was  not  able  to 
develop  a  tailored  data  collection  form  and  implement  a  data  collection  process. 
Thus,  we  did  not  collect  data  from  Creech  AFB. 

3.  Dyess  AFB 

a.  Supported  Missions 

The  Dyess  AFB  web  site  (Dyess  2006)  explains  the  mission  of  the 
7th  Bomb  Wing  (BW)  as  follows.  The  7th  BW  is  the  premier  operational  B-1B  unit 
in  the  U.S.  Air  Force.  It  has  the  capability  to  project  lethal  airpower  anywhere  in 
the  world  and  strike  at  a  moment’s  notice.  During  Operation  DESERT  FOX,  a 
Dyess  B-1  matched  with  a  B-1  from  Ellsworth  Air  Force  Base,  S.D.,  bombed  Iraqi 
Republican  Guard  barracks  at  Al-Kut,  Iraq,  with  500-pound  MK-82s  (Dyess 
2006).  This  first-ever  combat  bomb  drop  displayed  the  destructive  firepower  of 
the  B-1  and  echoed  the  battle  cry  “MORS  AB  ALTO,”  Latin  for  Death  From 
Above  (Dyess  2006).  The  B-1  carries  the  largest  payload  of  both  guided  and 
unguided  weapons  in  the  Air  Force  inventory  and  is  the  backbone  of  America’s 
long-range  bomber  force.  It  can  rapidly  deliver  massive  quantities  of  precision 
and  non-precision  weapons  against  any  adversary,  anywhere  in  the  world,  at  any 
time  (Factsheets  2006). 

There  are  six  flying  squadrons  located  at  Dyess  AFB,  Texas;  four 
B-1  squadrons  and  two  C-130  squadrons,  although  the  CWT  only  coordinated 
data  collection  with  the  B-ls  because  of  their  unique  mission.  The  generic  data 
collection  form  (Figure  5)  was  slightly  altered  to  adapt  to  the  B-1  mission.  Initial 
data  collection  revealed  the  need  to  adjust  wording  in  the  data  collection  form  to 
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make  it  more  understandable  to  aviators.  Data  collection  was  started  in 
November  and  continued  through  the  intensive  data  collection  period  of  9-13 
January  2006. 

b.  Data  Collection  Methods 

The  CWT  at  Dyess  AFB  coordinated  data  collection  with  the 
operations  officers  at  the  flying  squadrons.  The  flying  squadrons  agreed  to 
maintain  blank  data  collection  forms  at  the  squadron  operations  desk,  since  this 
is  one  of  the  locations  at  which  aviators  fill  out  paperwork  upon  completion  of 
their  missions.  CWT  personnel  entered  data  into  electronic  forms  and  forwarded 
them  to  NPS  for  data  analyses. 

4.  Nellis  AFB 

a.  Supported  Missions 

Several  different  types  of  operations  occur  simultaneously  at  Nellis 
AFB,  Nevada.  We  focused  data  collection  efforts  on  the  USAF  weapons  school 
(USAFWS),  air  warrior,  and  red  flag  because  their  schedule  is  regimented  and 
predictable.  The  mission  of  the  USAFWS  (Nellis  2006)  is  to  teach  graduate-level 
instructor  courses,  which  provide  the  world’s  most  advanced  training  in  weapons 
and  tactics  employment  to  officers  of  the  combat  air  forces.  Aircraft  flown  at  the 
USAFWS  include  A-10,  AC-130,  B-1,  B-2,  B-52,  F-15C,  F-15E,  F-16C,  F-117, 
HH-60,  MC-130,  and  MH-53  (Nellis  2006). 

Air  warrior  and  red  flag  are  exercises  conducted  at  the  414th 
combat  training  squadron  (Nellis  2006)  using  realistic  training  scenarios  that  are 
intended  to  develop  the  combat  readiness  and  survivability  of  participants. 
Combat  units  from  the  U.S.  and  several  allied  countries  engage  in  combat 
training  scenarios  conducted  within  the  Nellis  Range  Complex  (Nellis  2006). 

The  USAFWS  schedule  encompasses  both  academic  and  flying 
phases.  The  prime  opportunity  for  data  collection  is  during  the  flying,  or  mission 
execution  (ME),  phase.  The  ME  phase  lasts  one  week  with  several  flights  each 
day.  The  generic  data  collection  form  (Figure  5),  was  only  slightly  adjusted  to 


24 


accommodate  USAFWS  missions.  TAWS  is  heavily  used  at  the  USAFWS  as 
well  as  IRTSS,  so  the  IRTSS  questionnaire  in  Figure  6  was  also  used  at  Nellis 
AFB. 

b.  Data  collection  Methods 

Data  collection  at  the  USAFWS  was  coordinated  with  a  squadron 
commander  at  the  school  who  forwarded  all  the  aviators  an  electronic  copy  of  the 
data  collection  form  requesting/directing  support.  Completed  forms  were  sent  to 
NPS  for  analyses. 

Data  collection  forms  were  also  provided  to  weather  personnel 
supporting  both  air  warrior  and  red  flag.  These  weather  personnel  were  not 
stationed  at  Nellis  AFB,  but  rather  on  temporary  duty  (TDY)  for  the  purpose  of 
providing  weather  support  to  the  aviators  participating  in  the  two  exercises. 
Aviators  were  provided  paper  copies  of  data  collection  forms  prior  to  mission 
execution  and  completed  forms  were  sent  to  NPS  for  analyses. 

5.  Whiteman  AFB 

a.  Supported  Missions 

Whiteman  AFB,  Missouri  is  the  home  of  the  509th  Bomb  Wing, 
which  according  to  their  web  site  (Whiteman  2006)  operates  and  maintains  the 
Air  Force’s  premier  weapon  system,  the  B-2  bomber.  The  B-2  Spirit  is  a  multi¬ 
role  bomber  capable  of  delivering  both  conventional  and  nuclear  munitions.  A 
“dramatic  leap  forward  in  technology”  (Factsheets  2006),  the  bomber  represents 
a  major  milestone  in  the  U.S.  bomber  modernization  program.  The  B-2  brings 
“massive  firepower  to  bear,  in  a  short  time,  anywhere  on  the  globe  through 
previously  impenetrable  defenses”  (Factsheets  2006). 

Two  bombing  squadrons  are  located  at  Whiteman  AFB  and  data 
was  collected  from  both.  The  generic  data  collection  form  (Figure  5)  was 
significantly  shortened  for  use  at  Whiteman  AFB  because  the  aviators  place  the 
majority  of  their  emphasis  on  the  MEF  with  little  on  the  planning  weather 
forecasts  (PWFs). 
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b.  Data  Collection  Methods 

PWFs  are  provided  to  aircrews  telephonically  and  via  the  web, 
although  MEFs  are  briefed  in  person  approximately  three  hours  prior  to  takeoff. 
The  Whiteman  AFB  CWT  just  recently  integrated  weather  personnel  into  the 
flying  squadrons,  in  part  due  to  classification  issues  with  the  B-2.  Classification 
issues  also  result  in  restrictions  to  CWT  access  at  locations  in  which  the  aviators 
perform  debriefs.  Thus,  paper  copies  of  the  data  collection  form  were  placed  in 
the  debrief  area  at  the  maintenance  operations  squadron  and  then  faxed  to  the 
CWT  upon  completion. 

6.  PACAF  Units 

a.  Supported  Missions 

The  two  PACAF  bases  from  which  we  collected  data  were  Kunsan 
Air  Base  (AB)  and  Osan  AB,  both  located  in  South  Korea.  CWT  personnel  are 
well  integrated  into  the  flying  squadrons,  which  is  why  these  bases  were  asked  to 
participate. 

Kunsan  AB  is  located  on  the  western  side  of  the  South  Korean 
peninsula  bordered  by  the  Yellow  Sea,  approximately  150  miles  south  of  Seoul. 
Kunsan  AB  is  home  to  the  8th  Fighter  Wing  made  up  of  two  F-16  fighter 
squadrons  (Kunsan  2006).  The  F-16  Fighting  Falcon  is  a  compact,  multi-role, 
highly  maneuverable  fighter  aircraft  which  has  proven  itself  in  air-to-air  combat 
and  air-to-surface  attacks  (Factsheets  2006).  It  provides  a  relatively  low-cost, 
high-performance  weapon  system  for  the  U.S.  and  allied  nations  (Factsheets 
2006). 

Osan  AB  is  located  just  48  miles  south  of  the  Korean  DMZ  and  is 
home  to  the  51st  Fighter  Wing  “Mustangs”  (Osan  2006).  It  is  the  most  forward 
deployed  permanently-based  wing  in  the  Air  Force  and  provides  mission  ready 
airmen  to  execute  combat  operations  and  receive  follow-on  forces.  The  wing  has 
24  PAA,  F-16,  and  A-10  squadrons,  along  with  a  C-12  airlift  flight,  conducting  a 
full  spectrum  of  missions  in  defense  of  the  Republic  of  Korea  (Osan  2006).  The 
A/OA-10  Thunderbolt  II  (Factsheets  2006)  is  the  first  Air  Force  aircraft  specially 
designed  for  close  air  support  of  ground  forces.  The  aircraft  are  a  simple, 

26 


effective,  and  survivable  twin-engine  jet  that  can  be  used  against  all  ground 
targets,  including  tanks  and  other  armored  vehicles  (Factsheets  2006). 

b.  Data  Collection  Methods 

The  CWTs  at  Kunsan  AB  and  Osan  AB  did  not  have  the  time  to 
coordinate  applicable  questions  and  answer  options  on  the  data  collection  form, 
so  the  Nellis  AFB  data  collection  form,  including  the  TAWS  section,  was  used. 

CWT  personnel  are  fully  integrated  into  the  flying  squadrons  at 
Kunsan  AB,  available  four  to  five  hours  prior  to  takeoff  time  and  remaining  until 
the  last  aviator  steps  out  the  door  to  fly.  Paper  copies  of  the  data  collection  form 
were  completed  by  flight  leads  on  behalf  of  all  the  aircraft  in  their  flight,  consisting 
of  either  a  2-ship  or  4-ship  formation.  Completed  forms  were  mailed  to  NPS  for 
analyses. 

At  Osan  AB,  one  CWT  person  is  embedded  in  each  of  the  flying 
squadrons,  providing  mass  briefings  three  hours  prior  to  takeoff  and  updates 
(step  briefings)  one  hour  prior  to  takeoff.  The  Osan  AB  CWT  was  already 
collecting  feedback  from  flight  leads  in  the  form  of  a  paper  copy,  so  our  data 
collection  form  was  used  in  place  of  the  existing  one.  CWT  personnel  briefed 
flying  squadron  leadership  on  the  purpose  and  importance  of  the  form  to  help 
ensure  data  quality  and  quantity.  Completed  forms  were  mailed  to  NPS  for 
analyses. 

C.  NPS  AVIATOR  SURVEY 

Initial  feedback  from  the  ACC  and  PACAF  CWTs  indicated  the  data 
collection  form  was  too  lengthy,  which  might  negatively  affect  the  number  of 
forms  that  were  completed,  and  the  quality  of  the  data  on  the  forms  that  were 
completed.  To  improve  the  forms,  we  surveyed  AF  pilots  and  navigators 
attending  NPS.  We  also  conducted  subsequent  follow  up  surveys  and  interviews 
with  the  NPS  aviators  to  validate  and  interpret  the  results  from  our  analyses  of 
the  data  collected  from  the  ACC  and  PACAF  aviators. 

Although  most  NPS  aviators  have  only  been  in  a  non-flying  job  for  a  short 
time,  being  in  the  academic  environment  has  provided  them  the  opportunity  to 
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reflect  on  their  experiences.  Their  NPS  experiences  have  perhaps  also  given 
them  a  motivation  for  contributing  to  our  research  project,  since  they  too  were 
collecting  and  analyzing  data  for  their  thesis  research  projects.  Figure  7  is  the 
survey  emailed  to  the  36  AF  aviators  at  NPS,  most  of  who  have  attained  the  rank 
of  major,  have  at  least  nine  years  of  active  duty  AF  experience,  and  several 
hundred  hours  of  flying  experience. 

D.  SUMMARY  OF  DATA  COLLECTED 

1.  Aircrafts  and  Missions 

Data  was  collected  from  six  AF  bases  on  six  different  types  of  aircraft. 
Figure  8  gives  a  summary  of  the  missions  flown  during  the  data  collection 
timeframe,  including  average  number  of  missions  flown  per  day,  average  length 
of  each  mission,  and  the  type  of  mission  (e.g.,  air-to-air,  air-to-ground,  bombing). 
A  significant  number  of  the  missions  were  from  fighter  aircraft,  perhaps  because 
fighter  missions  are  shorter  than  bombing  and  reconnaissance  missions  and 
therefore  more  frequent. 

2.  Data 

Data  collection  was  initiated  in  November  2005  and  ended  in  January 
2006,  with  the  objective  of  having  data  collection  forms  completed  by  aviators, 
not  CWT  personnel.  Concentration  was  placed  on  collected  data  from  all  the 
bases  on  five  consecutive  days,  9-13  January  2006,  and  for  no  less  than  five 
consecutive  hours  each  day  (and  preferably  for  the  entire  day).  We  did  not 
attempt  to  collect  the  same  percentage  of  data  from  each  base  due  to  differences 
in  flight  operations  and  CWT  manning. 

E.  LIMITATIONS  OF  DATA  COLLECTION  PROCESS 

1.  Assumptions 

We  made  several  assumptions  regarding  or  data  collection  process.  First 
we  assumed  that  the  CWTs  were  provided  sufficient  guidance  on  how  to  get  to 
know  their  customer  prior  to  coordinating  their  data  collection  forms.  Second,  we 
assumed  that  the  CWTs  were  successful  at  tailoring  their  data  collection  forms  to 
develop  mission  appropriate  questions  and  answer  options.  Third,  we  assumed 
those  individuals  filling  out  the  data  collection  forms  fully  understood  the 
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questions  (especially  after  we  had  altered  the  initial  forms  in  response  to  aviator 
and  CWT  feedback).  Fourth,  we  assumed  the  aviators  providing  the  data  would 
provided  relatively  accurate  descriptions  and  assessments  of  the  forecasts  they 
received,  the  missions  changes  they  made  in  response  to  the  forecasts,  the 
weather  they  encountered  during  their  missions,  and  the  missions  changes  they 
made  in  response  to  the  weather  they  encountered, 

2.  Limitations 

A  major  shortcoming  was  the  limited  amount  of  data  collected  due  to  the 
limited  time  we  had  to  collect  data.  This  was  exacerbated  by  the  extended 
periods  of  holiday  leave  and  down  time  that  occurred  during  our  data  collection 
period.  We  also  relied  on  aviators  to  provide  accurate  assessments  of  weather 
forecasts,  weather  phenomena  encountered,  planning  changes,  and  weather 
impacts  during  execution.  The  quality  of  the  data  they  provided,  especially  the 
weather  data,  is  difficult  to  determine,  since  we  did  not,  and  in  general  were  not 
able  to,  verify  the  aviator  data  with  independently  collected  data.  In  some  cases 
security  limitations,  especially  at  Whiteman  AFB,  restricted  our  ability  to  collect 
data. 

F.  DATA  ANALYSIS  METHODOLOGY 

1.  Overview 

Hinz  (2004),  Jarry  (2005),  and  Butler  (2005)  used  a  method  in  which  they 
collected  weather  and  operator  data,  used  that  data  to  determine  forecast  and 
operator  performance  metrics,  and  then  compared  these  two  sets  of  metrics  to 
determine  operational  impacts  metrics.  This  method  is  schematically  depicted  in 
Figure  9  in  the  form  of  a  flow  chart  showing  the  data  collected  on  the  top  row, 
performance  analyses  in  the  middle  row,  and  the  assessment  of  operational 
impacts  on  the  bottom  row.  For  our  project,  all  the  data  was  provided  by 
aviators.  We  then  analyzed  the  aviator  data  to  assess  forecast  performance, 
aviator  performance,  and  the  impacts  of  the  forecast  and  the  actual  weather 
encountered  on  the  operational  performance  of  the  aviators.  The  results  of  our 
analyses  are  metrics  of  forecast  and  operational  performance,  and  the 
operational  impacts  of  the  forecasts  and  actual  weather  phenomena. 


29 


In  prior  studies,  such  as  Hinz  (2004),  Jarry  (2005),  and  Butler  (2005),  data 
was  collected  data  from  weather  personnel,  and  analyzed  by  weather  personnel. 
We  chose  to  work  with  data  obtained  directly  from  customers  because  we 
wanted  to  get  and  assess  the  value  of  direct  customer  feedback. 

2.  Goals 

The  focus  of  our  project  was  not  on  collecting  and  analyzing  large 
quantities  of  data,  but  rather  developing  a  process  for  collection  and  analyses. 
We  did  not  attempt  to  come  up  with  thorough  analyses  of  forecast  and 
operational  performance,  or  of  the  operational  impacts  of  forecasts  and  weather 
phenomena  encountered  during  mission  execution.  Our  goal  was  to  design, 
develop,  and  test  a  new  process  for  collecting  and  analyzing  customer  feedback. 
The  deliverable  product  we  set  out  to  create  was  a  prototype  system  capable  of 
efficiently  collecting  and  analyzing  customer  data  for  use  in  analyzing  forecast 
and  operational  performance,  and  the  operational  impacts  of  forecasts  and 
weather  phenomena  encountered  during  mission  execution. 

To  develop  such  a  prototype  system,  we  created  system  components  to 
perform  the  following  functions.  From  an  operational  perspective,  a  primary 
system  function  is  to  determine  what  operational  decisions  were  being  made 
based  on  PWFs  and  MEFs.  We  also  wanted  to  determine  key  timeframes  in 
which  aviators  use  weather  forecasts,  and  whether  or  not  aviators  felt  forecasts 
were  available  when  they  needed  them. 

From  a  meteorological  perspective,  a  primary  system  function  is  to 
determine  how  accurate  the  weather  forecasts  were,  the  probability  of  detecting 
negatively  impacting  weather  phenomena,  and  the  subsequent  false  alarm  rate 
for  negatively  impacting  weather  phenomena.  Previous  studies  determined 
these  parameters  from  the  perspective  of  weather  personnel.  Our  objective  was 
to  determine  them  from  the  perspective  of  aviators.  We  also  wanted  to 
determine  how  believable  the  weather  forecasts  were  to  the  aviators.  In  addition, 
we  wanted  to  identify  the  weather  phenomena  with  negative  operational  impacts 
that  were  most  commonly  misforecasted,  and  those  that  most  commonly  resulted 
in  unsuccessful  missions. 
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Overall,  our  main  goal  was  to  develop  a  prototype  system  for  analyzing  a 
combination  of  operational  and  weather  performance  data  provided  by  aviators  in 
order  to  assess  the  impacts  of  forecasts  and  weather  phenomena  on  the 
planning  and  execution  of  air  combat  missions.  One  of  our  main  motivations  was 
to  enable  CWTs  to  satisfy  several  requirements  in  AFMAN  15-129,  AFI  15-114, 
the  AFAA  report,  and  the  AFWSPV.  These  requirements  include  the 
development  of  a  user-friendly  system  for  measuring  the  operational  impacts  of 
weather  forecasts.  Once  our  prototype  is  tested,  it  will  be  adaptable  to  any  CWT 
in  AFW.  Our  prototype  will  also  be  adaptable  to  Naval  METOC  units  to  assist  in 
their  assessment  of  the  operational  impacts  of  METOC  products. 

An  additional  goal  was  to  support  the  developers  and  implementers  of 
TAWS  and  IRTSS  by  collecting  and  analyzing  TAWS  and  IRTSS  data.  New 
versions  of  TAWS  continue  to  be  fielded  and  because  of  feedback,  the  versions 
get  better  and  more  accurate.  Nellis  AFB  is  the  primary  active  duty  AFB  using 
IRTSS  and  since  Nellis  agreed  to  participate  in  this  project,  IRTSS  developers 
requested  our  support  in  obtaining  operational  feedback. 

3.  Metrics  Calculated 

The  main  results  of  our  analyses  were  five  quantitative  metrics  of  forecast 
performance  (items  a-e,  below),  and  15  quantitative  metrics  of  operational 
impacts  (items,  f-t,  below).  Several  of  these  metrics,  especially  those  of  forecast 
performance,  have  been  used  in  prior  studies  (e.g.,  Hinz  2004,  Jarry  2005,  Butler 
2005).  However,  our  calculation  of  these  metrics  was  based  on  a  different  type 
of  data  than  that  used  in  prior  studies.  The  main  difference  is  that  our  data:  (1) 
came  entirely  from  the  users  of  the  forecasts,  the  aviators,  not  from  weather 
personnel;  and  (2)  focused  exclusively  on  weather  phenomena  that  was 
forecasted  to  and/or  did  have  negative  impacts  on  the  missions.  Despite  these 
differences,  comparisons  of  the  two  types  of  metrics  are  useful,  and  are 
discussed  in  chapter  III. 

For  our  study,  we  focused  on  negatively  impacting  phenomena  in  the 

forecasts  and  in  the  pilot  report  observations.  Thus,  our  forecast  performance 

metrics  are  comparable  to,  for  example,  red  forecast  accuracy  metric  of  Hinz 
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(2004)  and  the  no  go  forecast  accuracy  of  Jarry  (2005).  In  our  case,  forecast 
performance  metrics  are  based  on  the  aviators  inferring  from  the  forecasts  the 
weather  phenomena  that  they  think  will  cause  negative  impacts,  and  then 
verifying  that  inference  when  they  execute  the  mission.  In  a  sense,  this  means 
that  we  treat  the  aviators  as  forecasters  because  we  rely  on  them  to  infer  the 
phenomena  that  will  have  negative  impacts,  and  also  as  verifiers  of  the  forecasts 
because  they  provide  the  observations  of  the  actual  weather.  Before  we 
calculated  forecast  performance  metrics  based  on  the  aviator  data,  we  checked 
that  the  time  and  location  of  the  forecasts  matched  the  time  and  location  of  the 
missions.  Only  the  data  from  missions  with  matching  times  and  locations  were 
used  in  the  calculation  of  those  metrics. 

a.  Forecast  Accuracy  for  Negatively  Impacting  Weather 
Phenomena 

Forecast  accuracy  (FAC)  is  the  percentage  of  forecasts  that  were 
accurate  (Hinz  2004,  Jarry  2005).  For  our  study,  FAC  is  the  number  of  missions 
for  which  negatively  impacting  weather  was  inferred,  and  actually  occurred,  by 
the  aviators,  divided  by  the  number  of  missions  for  which  negatively  impacting 
weather  was  inferred,  all  multiplied  by  100  to  come  up  with  a  percentage. 

b.  Probability  of  Detection  for  Negatively  Impacting 
Weather  Phenomena 

Probability  of  detection  (POD)  is  defined  as  a  verification  measure 
of  categorical  forecast  performance  equal  to  the  total  number  of  correct  event 
forecasts  divided  by  the  total  number  of  events  observed  (NOAA  2006).  For  our 
study,  POD  is  the  number  of  missions  for  which  a  specific  negatively  impacting 
phenomenon  (e.g.,  cloud  ceilings  below  thresholds)  was  inferred  by  the  aviators 
from  the  forecasts  and  was  also  observed  by  the  aviators,  divided  by  the  number 
of  missions  for  which  that  negatively  impacting  phenomenon  was  inferred  from 
the  forecasts  by  the  aviators,  all  times  100. 

c.  False  Alarm  Rate  for  Negatively  Impacting  Weather 
Phenomena 

The  NOAA  verification  glossary  (NOAA  2006)  defines  the  false 
alarm  rate  (FAR)  as  a  verification  measure  of  categorical  forecast  performance 
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equal  to  the  number  of  false  alarms  divided  by  the  total  number  of  event 
forecasts.  A  false  alarm  is  a  forecast  of  particular  weather  phenomena  that  does 
not  occur.  In  our  case,  if  an  aviator  inferred  from  the  forecast  that  cloud  ceilings 
would  have  a  negative  mission  impact,  but  did  not  actually  experience  this  on  the 
mission,  then  a  false  alarm  for  cloud  ceilings  occurred  for  that  mission.  We 
defined  FAR  as  the  number  of  missions  for  which  a  specific  negatively  impacting 
phenomenon  was  inferred  from  the  forecasts  by  the  aviators  but  did  not  actually 
occur,  divided  by  the  number  of  missions  from  which  that  negatively  impacting 
phenomenon  was  inferred  from  the  forecasts,  all  times  100. 

d.  Observed  Phenomena  with  Negative  Impacts  That 
Were/Were  Not  Forecasted 

This  metric  describes  the  percentage  of  missions  for  which  a 
specific  negatively  impacting  phenomenon  occurred  during  mission  execution 
and  whether  the  aviators  inferred  that  phenomenon  from  the  PWF  and/or  the 
MEF.  There  are  four  possibilities  for  this  metric:  (1)  phenomenon  occurred 
during  execution,  and  was  inferred  from  the  PWF  but  was  not  inferred  from  the 
MEF;  (2)  phenomenon  occurred  during  execution,  and  was  not  inferred  from  the 
PWF  but  was  inferred  from  the  MEF;  (3)  phenomenon  occurred  during  execution, 
and  was  inferred  from  the  PWF  and  was  inferred  from  the  MEF;  (4)  phenomenon 
occurred  during  execution,  and  was  not  inferred  from  the  PWF  and  was  not 
inferred  from  the  MEF.  This  metric  is  somewhat  similar  to  our  FAC  metric,  but 
with  a  focus  on  specific  phenomena,  and  without  accounting  for  the  total  number 
of  forecasts  that  were  correct  and  incorrect. 

e.  Percentage  of  Missions  for  Which  Negative  Mission 
Impacts  Occurred  and  Were/Were  Not  Indicated  by  the 
MEF 

This  metric  describes  the  percent  of  missions  that  incurred  negative 
weather  impacts  that  were  not  inferred  from  the  PWF  or  the  MEF.  This  is  similar 
to  other  metrics  mentioned  above  but  provides  a  more  general  assessment  than 
the  prior  metrics. 
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f.  Negative  Mission  Impacts  Inferred  from  Planning 
Weather  Forecast  and  MEF 

This  metric  describes  the  number  and  percentage  of  missions  for 
which:  (1)  the  aviators  inferred  from  the  PWF  and  the  MEF  that  a  specific 
negative  impact  would  occur;  (b)  a  specific  negative  impact  occurred  during 
execution.  By  specific  negative  impacts,  we  mean  impacts  such  as  scheduling 
changes,  weapons  changes,  inability  to  takeoff,  etc. 

g.  Mission  Plan  Changes  Made  Due  to  Planning  Weather 
Forecast  and  MEF 

This  metric  describes  the  number  of  missions  for  which  aviators 
made  mission  plan  changes  base  on  negatively  impacting  weather  phenomena 
being  inferred  from  the  planning  weather  forecast  and/or  MEF. 

h.  Mission  Plan  Changes  Due  to  Negative  Mission  Impacts 

This  metric  describes  the  number  of  times  a  mission  plan  is 

changed  due  to  aviators  inferring  negative  mission  impacts  in  forecasts. 

/.  Weather  Phenomena  Resulting  in  Unsuccessful  Mission 

This  metric  describes  the  percentage  of  missions  deemed 
unsuccessful  by  aviators  due  to  negative  impacts  from  weather  phenomena. 

j.  Planning  Weather  Forecast  Positive  Mission 
Contribution 

This  metric  describes  the  percent  of  missions  for  which  a  positive 
mission  impact  (PMC)  is  made  by  the  planning  weather  forecast.  If  aviators  infer 
a  negative  mission  impact  from  the  planning  weather  forecast,  make  mission 
plan  changes,  and  then  achieve  their  primary  mission,  the  PMC  criteria  are  met. 

k.  MEF  Positive  Mission  Contribution 

This  metric  describes  the  percent  of  missions  for  which  a  positive 
mission  impact  (PMC)  is  made  by  the  planning  weather  forecast.  If  aviators  infer 
a  negative  mission  impact  from  the  planning  weather  forecast,  make  mission 
plan  changes,  and  then  achieve  their  primary  mission,  the  PMC  criteria  are  met 
for  our  study. 
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l.  Potential  Positive  Mission  Contribution 

This  metric  describes  the  percent  of  missions  for  which  a  positive 
mission  contribution  (PPMC)  is  made  by  the  planning  weather  forecast  or  the 
MEF.  Our  study  defines  a  PPMC  as  an  accurate  forecast  is  issued,  aviators 
make  no  mission  plan  changes  in  response,  and  the  primary  mission  is  not 
accomplished. 

m.  Planning  Weather  Delay  Effects  on  Mission 
Accomplishment 

This  metric  describes  the  percent  of  unsuccessful  missions  where 
the  aviator  would  have  checked  the  planning  weather  forecast  earlier  had  it  been 
available. 

n.  Negative  MEF,  No  Mission  Changes 

This  metric  describes  the  percent  of  missions  that  are  successful 
even  though  negative  weather  impacts  are  forecasted  in  the  MEF,  but  the  aviator 
does  not  make  any  mission  plan  changes  due  to  the  MEF. 

o.  Mission  Plan  Changes  Due  to  Planning  Weather  and 
MEF 

This  metric  describes  the  number  of  times  aviators  make  a  mission 
plan  change  due  to  the  planning  weather  forecast  and  then  change  the  same 
mission  plan  after  seeing  the  MEF. 

p.  Unsuccessful  Mission  Due  to  Inaccurate  MEF 

This  metric  describes  the  percent  of  missions  deemed  unsuccessful 
by  aviators  where  the  aviator  feels  the  MEF  is  inaccurate. 

q.  Weather  Phenomena  Actually  Occurring  Resulting  in 
Mission  Success/Failure 

This  metric  describes  the  percent  of  each  weather  phenomena 
actually  occurring  during  the  mission  and  then  whether  or  not  the  aviators 
deemed  the  mission  successful/unsuccessful. 

r.  Planning  Weather  Needs 

This  metric  describes  the  percent  of  missions  evaluated  where  the 
aviator  would  have  checked  the  planning  weather  earlier  had  it  been  available. 
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s.  Timeframe  of  MEF  Usage 

This  metric  describes  the  number  of  hours  prior  to  mission  takeoff 
that  aviators  check  the  MEF. 

t.  Successful  Missions 

This  metric  describes  the  percent  of  missions  considered 
successful  by  the  aviator  even  though  they  did  not  accomplish  what  they 
intended  to  accomplish  on  the  mission. 

Each  different  data  collection  form  required  a  separate  data 
analysis  worksheet  based  on  the  initial  coordination  with  aviators.  For  this 
reason,  some  the  metrics  listed  above  will  not  be  calculated  for  every  base. 
Figure  10  is  a  description  of  calculations  performed  to  analyze  ACC  and  PACAF 
data,  plus  instructions  for  displaying  the  results  of  the  analyses.  Figure  10  is 
based  on  data  collected  from  Kunsan  AB,  Nellis  AFB,  and  Osan  AB.  Data 
analyses  for  the  other  bases  will  be  abbreviated  versions  of  Figure  10  because 
data  collection  was  not  as  robust. 

4.  Online  Data  Collection  and  Analysis  System 

a.  Software 

To  be  most  effective,  assessment  of  forecast  performance  and  the 
operational  impacts  of  forecasts  must  be  timely.  Hinz  (2004)  and  Jarry  (2005) 
both  had  large  lag  times  between  their  final  collections  of  data  from  operational 
units  until  their  final  metrics  reports  were  completed.  Butler  (2005)  found  that  for 
a  metrics  program  to  be  effective,  the  turn  around  time,  from  collection  of  data  to 
delivery  of  metrics  results,  must  be  near-real  time.  This  can  be  done  by 
leveraging  information  technology  (IT),  which  is  just  what  Butler  (2005)  did. 

Butler  (2005)  used  three  computer  languages,  Hypertext  Markup 
Language  (HTML),  PHP:  Hypertext  Preprocessor  (PHP),  and  My  Structured 
Query  Language  (MySQL),  to  allow  users  to  enter  data  via  the  internet  into  a 
database  located  at  NPS,  have  the  data  analyzed,  and  then  receive  results  in 
near  real-time.  For  in-depth  explanation  of  this  process,  see  Butler’s  (2005) 
Chapter  II.  The  bottom  line  is  that  by  leveraging  IT,  a  lag  time  of  six  months  can 
be  decreased  to  a  few  minutes. 
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b.  Flow  of  Data 

For  our  system,  an  adaptation  of  the  Butler  (2005)  system,  once 
data  is  entered  into  the  database,  it  is  analyzed  by  the  IT  software  and  results 
are  provided  to  the  user  in  both  numerical  and  graphical  form.  Figure  11  shows 
the  flow  of  data,  from  collection  by  aviators,  to  analyses  at  NPS,  to  near-real 
reporting  of  results.  Aviators  were  able  to  input  their  data  directly  into  the  online 
data  collection  form,  an  example  of  which  is  given  in  Figure  12  for  Osan  AB. 
Data  analysis  worksheets  were  developed  to  manipulate  the  aviator  feedback 
received  via  the  data  collection  forms.  Most  of  the  analyses  were  automated  and 
completed  via  the  online  system.  Further  explanations  of  the  data  flows  and 
analyses  can  be  found  in  Butler’s  (2005)  Chapter  II. 
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III.  RESULTS 


A.  OVERVIEW 

One  of  the  three  goals  of  this  study  was  to  adapt  an  existing  web-based 
system  from  Butler  (2005)  for  data  collection  and  analysis  for  Air  Force  use  at  the 
CWT  level.  The  adapted  system  we  created  can  be  found  at 
http://wx.met.nps.navy.mil/metrics.  The  second  goal  was  to  implement  and  test 
the  system,  and  analyze  the  results.  The  third  goal  was  to  use  results  to  provide 
advice  to  AFW,  ACC,  and  PACAF  on  how  to  alter  their  procedures  and  identify 
improvement  opportunities.  This  chapter  focuses  on  the  third  goal  by  discussing 
the  data  we  collected  and  the  results  of  our  data  analyses.  The  most  pertinent 
results  are  presented  in  this  report.  Additional  results  fount  at 

http://wx.met.nps.navv.mil/metrics/airforce/accu.html.  Since  the  primary  focus  of 
this  research  was  on  the  process  and  not  the  collection  and  analysis  of  an 
abundance  of  data,  only  preliminary  conclusions  about  forecast  performance  and 
the  operational  impacts  of  forecasts  can  be  determined.  However,  our  results  do 
allow  us  to  make  a  thorough  assessment  of  our  main  goal,  the  development, 
testing,  and  operational  application  of  an  online  prototype  metrics  system. 

After  data  collection  began,  initial  answers  implied  aviators  misunderstood 
some  of  the  questions  in  the  form.  For  example,  some  completed  data  collection 
forms  indicated  aircrews  were  providing  feedback  on  more  than  one  mission  per 
form,  and  it  wasn’t  clear  which  answers  pertained  to  each  mission,  rendering  the 
data  unusable.  Therefore,  revisions  were  made  to  the  wording  of  some 
questions.  Collection  resumed  and  data  quality  and  quantity  increased.  Overall, 
approximately  nine  percent  of  the  data  collection  forms  were  suspect  and 
consequently  left  out  of  analyses. 

Data  of  sufficient  quality  to  be  used  in  our  data  analyses  was  collected  for 
a  total  of  107  missions  during  the  intensive  data  collection  period  of  9-13  January 
2006.  As  noted  in  chapter  II,  this  figure  of  107  missions  is  probably  an 
underestimate  of  the  actual  number  of  aircraft  that  flew,  because  in  some  cases 
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multiple  flights  that  were  part  of  the  same  mission  were  reported  on  one  data 
collection  form.  Of  the  107  missions,  40  percent  were  flown  out  of  the  two 
participating  bases  in  PACAF.  Figure  13  is  a  summary  of  data  collected, 
categorized  by  air  base  (rows)  and  air  frame  (columns).  The  summary  describes 
the  percent  of  total  missions  performed  by  each  base,  the  percentage  of 
squadron  missions  completing  data  collection  forms,  and  to  which  major 
command  (ACC  or  PACAF)  each  base  was  assigned.  The  CWT  at  Creech  AFB 
was  never  able  to  collect  data  with  all  of  the  organizational  transitions  and  other 
challenges  they  faced. 

Planning  weather  forecasts  (PWFs)  were  defined,  for  this  study,  as 
forecasts  issued  prior  to  six  hours  in  advance  of  takeoff  time,  while  forecasts 
within  six  hours  of  takeoff  were  referred  to  as  mission  execution  forecasts 
(MEFs).  When  aviators  were  asked  the  earliest  time  prior  to  mission  takeoff  they 
first  looked  at  the  PWF,  if  available,  the  overwhelming  response  was  12-24  hours 
prior.  The  MEF  was  reviewed  between  one  and  three  hours  prior  to  mission 
takeoff. 

We  found  that  terms  used  in  the  data  collection  form  may  indicate  multiple 
definitions  at  differing  AF  bases.  For  example,  the  PACAF  data  indicated  that 
the  term  “mission  successful”  had  several  meanings.  We  concluded  that,  in 
general,  if  aviators  accomplished  something  useful,  but  not  necessarily  what  they 
intended  to  accomplish,  the  mission  was  deemed  successful  by  the  aviators. 
Results  indicated  82  percent  of  all  missions  analyzed  were  deemed  successful 
by  aviators,  while  only  seven  percent  were  considered  unsuccessful,  and  for  11 
percent  of  the  missions  success  was  not  assessed  by  the  aviators. 

B.  PERFORMANCE  AND  IMPACTS  RESULTS 

1.  Forecast  Accuracy  Results 

The  FAC  was  calculated  for  all  weather  phenomena  that  actually  occurred 
during  the  missions  and  led  aviators  to  infer  negative  mission  impacts.  Figure  14 
shows  the  FAC  for  forecasts  of  phenomena  (listed  on  the  horizontal  axis)  from 
which  aviators  inferred  negative  mission  impacts.  PWF  FAC  is  shown  in  blue, 

MEF  FAC  in  purple,  and  the  average  of  PWF  and  MEF  FAC  in  yellow.  It  is 
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important  to  note  large  values  of  FAC  for  a  given  phenomenon  are  generally 
associated  with  a  very  small  sample.  The  FAC  for  turbulence,  and  contrails  was 
zero  because  these  weather  phenomena  were  never  inferred  from  the  forecasts 
or  observed.  The  FAC  for  in-flight  icing  and  icing  run  times  was  zero  because 
the  aviators  reported  that  the  never  accurately  inferred  these  weather 
phenomena  from  the  forecasts  but  did  observe  them.  This  may  mean  the 
forecasts  were  inaccurate,  that  the  aviators  did  not  accurately  assess  the 
forecast,  or  that  they  did  not  accurately  infer  the  negative  impacts  of  the 
forecasted  phenomena.  It  is  also  possible  that  the  forecast  was  accurate  but  that 
mission  plan  changes  were  made  that  led  to  the  unexpected  negative  impacts. 
These  could  include  small  changes  in  mission  timing  and/or  location,  weapons, 
tactics,  etc.). 

The  PWF  FAC  shows  100  percent  accuracy  in  forecasting  visibility  aloft. 
This  may  be  misleading  because  visibility  aloft  actually  occurred,  and  had  a 
negative  impact  on  the  mission  as  deemed  by  aviators,  in  only  four  missions. 
The  mean  PWF  FAC  for  cloud  ceiling,  cloud  layers,  surface  visibility,  and  visibility 
aloft  is  69.2  percent  with  a  standard  deviation  of  16.9  percent  between  negatively 
impacting  weather  phenomena.  The  highest  MEF  FAC  was  80  percent  for 
visibility  aloft.  The  mean  MEF  FAC  for  cloud  ceiling,  cloud  layers,  surface 
visibility,  and  visibility  aloft  was  69.2  percent  with  a  standard  deviation  of  8.9 
percent  between  negatively  impacting  weather  phenomena.  In  general,  the  PWF 
FAC  and  the  MEF  FAC  were  similar. 

The  mean  FAC  for  both  PFW  and  MEF  was  69.2  percent,  which  is 
comparable  to  the  mean  No  Go  MEF  forecast  accuracy  in  previous  studies  (Hinz 
2004,  Jarry  2005).  For  example,  Jarry  (2005)  found  the  mean  No  Go  MEF  FAC 
for  AMC  data  during  fiscal  year  2004  was  65.6  percent.  This  suggests  aviators 
are  assessing  the  forecast  accuracy  roughly  the  same  as  weather  personnel  who 
use  instrumentation  for  verification. 
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2.  Probability  of  Detection  for  Negatively  Impacting  Weather 
Phenomena 

The  POD  was  calculated  for  all  weather  phenomena  that  actually  occurred 
during  the  mission  and  led  aviators  to  infer  negative  mission  impacts.  Figure  15 
shows  the  POD  for  forecasts  of  phenomena  (listed  on  the  horizontal  axis)  from 
which  aviators  inferred  negative  mission  impacts.  The  blue  depicts  PWF  POD 
while  the  purple  depicts  MEF  POD.  It  is  important  to  note  large  values  of  POD 
for  a  given  phenomenon  are  generally  associated  with  a  very  small  sample  size. 
The  POD  for  turbulence,  and  contrails  was  zero  because  these  weather 
phenomena  were  never  inferred  from  the  forecasts  or  observed.  The  POD  for  in¬ 
flight  icing  and  icing  run  times  was  zero  because  the  aviators  reported  that  they 
never  accurately  inferred  these  weather  phenomena  from  the  forecasts  but  did 
observe  them.  This  may  mean  the  forecasts  were  inaccurate,  that  the  aviators 
did  not  accurately  assess  the  forecast,  or  that  they  did  not  accurately  infer  the 
negative  impacts  of  the  forecasted  phenomena.  It  is  also  possible  that  the 
forecast  was  accurate  but  that  mission  plan  changes  were  made  that  led  to  the 
unexpected  negative  impacts.  These  could  include  small  changes  in  mission 
timing  and/or  location,  weapons,  tactics,  etc.). 

Overall,  the  PWF  and  MEF  both  had  the  highest  POD  for  surface  visibility 
at  86.7  percent.  The  mean  POD  for  PWFs  was  65.5  percent  with  a  standard 
deviation  of  15.9  percent  where  the  PWFPOD  for  surface  visibility  was  86.7 
percent  while  the  PWFPOD  for  cloud  layers  was  50  percent.  The  mean  POD  for 
MEFs  was  65.5  percent  with  a  standard  deviation  of  15.1  percent,  which  is 
comparable  to  the  mean  No  Go  MEFPOD  in  previous  studies.  The  MEFPOD  for 
surface  visibility  was  86.7  percent  while  the  MEFPOD  for  cloud  ceilings  was  53.3 
percent.  Jarry  (2005)  found  the  mean  No  Go  MEFPOD  for  the  AMC  fiscal  year 
2004  to  be  67.2  percent. 

3.  False  Alarm  Rate  for  Negatively  Impacting  Weather 
Phenomena 

The  FAR  was  calculated  for  all  weather  phenomena  that  actually  occurred 

during  the  mission  and  led  aviators  to  infer  negative  mission  impacts.  Figure  16 

shows  the  FAR  for  forecasts  of  phenomena  (listed  on  the  horizontal  axis)  from 
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which  aviators  inferred  negative  mission  impacts.  The  PWFFAR  is  shown  in  blue 
and  the  MEFFAR  in  purple.  The  FAR  for  turbulence,  and  contrails  was  zero 
because  these  weather  phenomena  were  never  inferred  from  the  forecasts  or 
observed.  The  FAC  for  in-flight  icing  and  icing  run  times  was  100  because  the 
aviators  reported  that  the  never  accurately  inferred  these  weather  phenomena 
from  the  forecasts  but  did  observe  them.  This  may  mean  the  forecasts  were 
inaccurate,  that  the  aviators  did  not  accurately  assess  the  forecast,  or  that  they 
did  not  accurately  infer  the  negative  impacts  of  the  forecasted  phenomena.  It  is 
also  possible  that  the  forecast  was  accurate  but  that  mission  plan  changes  were 
made  that  led  to  the  unexpected  negative  impacts.  These  could  include  small 
changes  in  mission  timing  and/or  location,  weapons,  tactics,  etc.). 

The  mean  PWFFAR  was  31.2  percent  with  a  standard  deviation  of  36.7 
percent.  The  mean  MEFFAR  was  52.1  percent  with  a  standard  deviation  of  37.7 
percent.  These  values  are  similar  to  previous  studies  as  Jarry  (2005)  found  the 
mean  No  Go  MEFFAR  for  the  AMC  fiscal  year  2004  MEF  verification  data  was 
34.4  percent  with  a  standard  deviation  between  13  CWTs  of  33.3  percent.  As 
previously  mentioned,  metrics  from  the  two  studies  cannot  be  directly  compared 
since  our  data  was  aviator-assessed  and  previous  studies  were  assessed  by 
weather  personnel  using  meteorological  equipment.  It  is,  although,  interesting  to 
note  the  comparisons. 

4.  Negative  Impacts  Inferred  and  Mission  Plan  Changes 

In  36  percent  of  the  total  missions,  PWFs  were  inferred  by  aviators  to 
have  negative  mission  impacts,  but  in  only  21  percent  of  the  missions  did 
aviators  change  mission  plans  due  to  the  PWFs.  In  39  percent  of  the  total 
missions,  MEFs  were  inferred  by  aviators  to  have  negative  mission  impacts,  but 
in  only  21  percent  of  the  missions  did  aviators  change  mission  plans  after 
interpreting  the  MEF.  Even  though  aviators  only  changed  mission  plans  in 
response  to  forecasts  in  21  percent  of  the  missions,  results  indicate  36  percent  of 
all  missions  experienced  weather  phenomena  with  negative  mission  impacts,  as 
deemed  by  aviators.  This  raises  the  question  why  aviators  did  not  change  their 
mission  plans  more  often.  Perhaps  changing  mission  plans  was  not  an  option  or 
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perhaps  the  aviators  to  take  their  chances  and  hope  for  weather  that  was 
sufficiently  benign  to  allow  them  to  complete  their  mission  or  alternate  mission. 
This  might  have  been  the  case  if  the  aviators  did  not  have  confidence  in,  and 
therefore  ignored,  the  forecasts,  or  if  they  had  a  back-up  plan  based  on  the 
negative  mission  impacts  they  inferred.  Despite  the  reported  negative  impacts, 
the  aviators  assessed  82  percent  of  all  their  missions  as  successful. 

5.  Observed  Phenomena  with  Negative  Impacts  That  Were/Were 
Not  Forecasted 

Figure  17  shows  the  percent  of  missions  that  experienced  phenomena  (on 
the  horizontal  axis)  with  negative  impacts  that  were/were  not  forecasted  in  the 
PWF  and/or  the  MEF.  Results  indicate  that  in  six  percent  of  all  missions,  the 
PWF  was  interpreted  to  contain  negatively  impacting  weather  phenomena  when 
the  MEF  did  not,  and  for  six  percent  of  all  missions  the  MEF  forecasted 
negatively  impacting  weather  when  the  PWF  did  not.  Overall,  the  PWF  and  MEF 
appear  to  be  equally  accurate,  although  the  PWF  did  better  at  cloud  ceilings 
while  the  MEF  was  better  at  cloud  layers.  When  turbulence  occurred  and 
negatively  impacted  the  mission,  it  was  not  contained  in  either  the  PWF  or  MEF. 
This  may  imply  either  the  forecasts  were  inaccurate  or  the  aviator  did  not 
interpret  the  forecast  accurately.  Of  the  three  weather  phenomena  having  the 
largest  number  of  negative  impacts  to  missions,  the  forecasts  were  most 
accurate  for  surface  visibility,  although  surface  visibility  also  had  the  highest  FAR 
(see  Figure  16).  Additionally,  of  the  top  three  weather  phenomena  having 
negative  impacts  to  missions,  the  forecasts  did  the  poorest  job  at  forecasting 
cloud  layers,  which  indicates  this  is  an  area  deserving  of  further  attention  (e.g., 
further  research,  education  and  training,  etc.). 

6.  Percentage  of  Missions  for  Which  Negative  Mission  Impacts 
Occurred  and  Were/Were  Not  Indicated  by  the  MEF 

Figure  18  shows  the  percent  of  all  missions  that  experienced  negative 
impacts  (on  the  horizontal  axis)  that  were/were  not  indicated  by  the  MEF.  Low 
level  training  and  instrumentation  training  had  the  lowest  chance  of  being 
indicated  or  forecasted  by  the  MEF  which  may  mean  that  the  aviators  did  not 
interpret  these  negative  mission  impacts  correctly  or  the  MEFs  did  not  accurately 
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depict  the  weather  phenomena.  It  may  also  mean  the  weather  was  too  good  to 
accomplish  instrumentation  training  because  overcast  conditions  were  necessary 
and  the  schedule  changed  after  the  aviator  looked  at  the  MEF  and  the  weather 
ended  up  improving.  Therefore,  the  aviators  could  no  longer  accomplish 
instrumentation  training,  not  because  of  forecast  or  interpretation  inaccuracies, 
but  because  the  schedule  changed  out  of  their  control.  Additional  information  on 
why  the  mission  was  not  completed  would  have  helped  as  there  is  not  enough 
data  to  make  a  determination. 

In  80  percent  of  the  missions  where  the  aviators  could  not  see  the  target, 
they  had  been  pre-warned  by  the  MEF.  Of  the  missions  that  could  not  takeoff  as 
scheduled,  the  MEF  was  65  percent  accurate  at  identifying  this  negative  impact. 
When  aviators  were  not  able  to  perform  air-to-air  training,  forecasts  indicated  this 
impact  every  time. 

7.  Negative  Mission  Impacts  Inferred  From  Planning  Weather 
Forecast  and  MEF 

Figure  19  shows  the  percent  of  missions  for  which  mission  plan  changes 
(on  the  horizontal  axis)  were  made  by  aviators  in  response  to  negatively 
impacting  weather  phenomena  indicated  by  only  the  PWF  (light  blue),  only  the 
MEF  (dark  blue),  or  both  the  PWF  and  the  MEF  (green).  Although  these  results 
may  include  multiple  plan  changes  in  response  to  either  the  MEF  or  PWF,  the 
most  significant  finding  here  is  that  26  percent  of  mission  plan  changes  were 
made  by  aviators  in  response  solely  to  the  PWF  while  only  20  percent  were 
changed  solely  in  response  to  the  MEF.  Aviators  made  aircrew  changes  in  12 
percent  of  the  missions,  weapon  changes  in  17  percent  of  the  missions,  schedule 
changes  in  17  percent  of  the  missions,  and  mission  plan  changes  in  11  percent 
of  the  missions.  Another  item  to  note  is  that  mission  plan  changes  (i.e.  low-level 
versus  high-level,  strike  target  on  the  ground  versus  air-to-air  training,  etc.)  were 
only  made  in  response  to  the  PWF.  This  clearly  indicates  that  CWTs  need  to 
place  more  emphasis  on  PWFs. 
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8.  Negative  Impacts  Inferred  From  PWF 

Figure  20  shows  the  number  of  missions  for  which  indicated  negative 
impacts  (colored  symbols)  were  inferred  by  aviators  from  PWFs  of  indicated 
weather  phenomena  (horizontal  axis).  Note  that  the  forecasted  phenomena 
associated  with  the  largest  number  of  inferred  negative  mission  impacts  were 
cloud  ceilings  (associated  with  inability  to  see  target),  cloud  layers  (associated 
with  inability  to  see  target),  and  surface  visibility  (associated  with  inability  to 
takeoff  or  land). 

Aviators  inferred  from  12  PWFs  that  they  would  not  be  able  to  takeoff  or 
land  due  to  surface  visibility  negatively  impacting  their  mission.  From  an  AFW 
perspective,  it  is  valuable  to  see  from  Figure  14  that  the  PWF  FAC  for  surface 
visibility  was  nearly  70  percent.  Aviators  assessed  22  PWFs  as  having  negative 
impacts  due  to  surface  visibility  and  12  PWFs  as  having  negative  impacts  due  to 
cloud  layers  and  cloud  ceilings,  although  this  could  be  misleading  if  aviators 
inferred  multiple  negative  impacts  due  to  the  same  weather  phenomena. 

9.  Mission  Plan  Changes  Made  Due  to  PWF 

Figure  21  depicts  the  number  of  missions  for  which  indicated  mission  plan 
changes  (colored  symbols)  were  made  based  on  PWFs  of  indicated  weather 
phenomena  (horizontal  axis).  Note  that  the  mission  schedule  was  changed  for 
four  missions  in  response  to  the  PWF  of  surface  visibility.  If  aviators  made 
mission  plan  changes  every  time  they  inferred  a  negative  mission  impact  would 
occur,  Figure  21  would  show  at  least  the  same  number  of  mission  plan  changes 
for  each  weather  phenomena  as  Figure  20  shows  for  negative  impacts.  This 
may  be  a  bit  confusing  as  multiple  negative  mission  impacts  can  be  inferred  from 
one  weather  phenomena,  and  multiple  mission  plan  changes  can  be  made  from 
that  same  phenomena.  Surface  visibility,  for  instance,  was  deemed  to  have 
negative  mission  impacts  on,  at  the  very  least,  12  missions,  but  aviators  did  not 
make  any  mission  plan  changes  for  10  of  those  missions.  Note  that  in  planning 
two  of  the  missions,  aviators  made  mission  plan  changes  even  though  no 
weather  phenomena  were  inferred  to  have  negative  mission  impacts.  This 
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suggests  aviators  did  not  believe  the  forecasts,  or  chose  to  give  higher  priority  to 
factors  other  than  the  forecasts  in  planning  their  missions. 

10.  Negative  Impacts  Inferred  From  MEF 

Figure  22  shows  the  number  of  missions  for  which  indicated  negative 
impacts  (colored  symbols)  were  inferred  by  aviators  from  MEFs  of  indicated 
weather  phenomena  (horizontal  axis).  Note  that  the  inability  to  takeoff  or  land 
was  inferred  for  14  missions  based  on  forecasts  of  surface  visibility  in  the  MEFs 
for  those  missions.  Surface  visibility  once  again  was  assessed  to  have  the  most 
negative  mission  impacts  on  takeoffs  and  landings,  and  negative  impacts  from 
this  phenomenon  were  inferred  in  14  MEFs.  Negative  mission  impacts  were 
inferred  for  a  total  of  23  missions  due  to  surface  visibility  in  the  MEF,  and  cloud 
layers  and  clouds  ceilings  had  the  next  highest  inference  of  negative  mission 
impacts  at  14  each.  The  MEFFAC  for  surface  visibility  is  just  over  60  percent. 
The  following  section  will  discuss  whether  or  not  aviators  made  mission  plan 
changes  in  response  to  the  forecasts.  It  is  important  to  note  that  aviators  do  not 
always  have  the  ability  to  make  mission  plan  changes  in  response  to  the 
forecasts. 

11.  Mission  Plan  Changes  Made  Due  to  MEF 

Figure  23  depicts  the  number  of  missions  for  which  indicated  mission  plan 
changes  (colored  symbols)  were  made  based  on  MEFs  of  indicated  weather 
phenomena  (horizontal  axis).  As  an  example  from  this  figure:  weapon  changes 
were  made  for  four  missions  based  on  the  cloud  layer  forecasts  in  the  MEFs  for 
those  missions.  If  aviators  made  mission  plan  changes  every  time  they  inferred 
a  negative  mission  impact  would  occur,  Figure  23  would  show  at  least  the  same 
number  of  mission  plan  changes  for  each  weather  phenomena  as  Figure  22 
shows  for  each  negative  impact.  We  focus  on  surface  visibility  once  again,  as  it 
was  deemed  to  have  negative  mission  impacts  on,  at  the  very  least,  14  missions 
(Figure  22),  but  aviators  did  not  make  any  mission  plan  changes  for  10  of  those 
missions  (Figure  23),  suggesting  they  did  not  believe  the  forecast  for  at  least  four 
missions.  Results  also  indicate  cloud  ceilings  and  cloud  layers  were  inferred  to 
have  negative  mission  impacts  on  at  least  1 1  missions  each  (Figure  22)  and  no 
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mission  plan  changes  were  made  during  five  of  those  1 1  missions  (Figure  23). 
This  suggests  cloud  ceiling  and  cloud  layer  forecasts  were  believed  less  than 
surface  visibility  forecasts. 

12.  Weather  Phenomena  Resulting  in  Unsuccessful  Mission 

After  seeing  how  aviators  interpret  forecasts  and  then  make  mission  plan 
changes,  it  is  important  to  highlight  which  weather  phenomena  caused  the  small 
percentage  of  unsuccessful  missions.  Figure  24  shows  the  percent  of  missions 
deemed  unsuccessful  due  to  negative  impacts  from  weather  phenomena 
(horizontal  axis).  Seven  percent  of  all  missions  analyzed  were  deemed 
unsuccessful  by  aviators.  Five  percent  of  all  mission  failures  were  due  to  surface 
visibility,  and  two  percent  were  due  to  visibility  aloft.  Results  from  Figures  20  and 
21  suggest  aviators  did  not  believe  surface  visibility  forecasts  during  four 
missions.  If  aviators  had  made  mission  plan  changes  each  time  the  forecast 
indicated  negative  mission  impacts,  the  outcome  might  have  been  different. 
Results  also  show  the  MEF  FAR  for  surface  visibility  was  nearly  40  percent, 
which  may  support  the  reason  aviators  did  not  make  mission  plan  changes. 

13.  Positive  Mission  Contribution  to  Successful  Missions 

Figure  25  depicts  the  percent  of  missions  for  which  a  positive  mission 
contribution  (PMC)  was  made  by  the  PWF  (seen  in  blue)  or  MEF  (seen  in 
purple).  PMC  criteria  are  summarized  in  the  blue  text  box  within  the  figure  and 
are  described  in  detail  in  Chapter  II.  Percentages  are  based  on  the  number  of 
missions  deemed  successful  by  aviators.  For  all  bases,  10.2  percent  of 
successful  missions  received  a  PMC  from  their  PWFs,  and  10.2  percent  of 
successful  missions  received  a  PMC  from  their  MEFs.  The  PMC  for  Beale  AFB, 
Nellis  AFB,  and  Whiteman  AFB  was  zero  because  there  was  little  negatively 
impacting  weather  phenomena  identified  in  the  forecasts  by  aviators,  thus  the 
opportunity  to  have  a  PMC  was  extremely  low.  The  greatest  PMC  to  mission 
success  was  Osan  AB,  where  the  PWF  and  MEF  each  had  roughly  a  six  percent 
PMC  to  all  successful  missions.  As  mentioned  above,  Osan  AB  also  flew  40 
percent  of  all  missions  analyzed. 
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14.  Potential  Positive  Mission  Contribution  to  All  Missions 

Figure  26  shows  the  percent  of  all  missions  for  which  a  potential  positive 
mission  contribution  (PPMC)  was  made  by  the  PWF  (seen  in  blue)  or  the  MEF 
(seen  in  purple).  PPMC  criteria  are  summarized  in  the  blue  text  box  within  the 
figure  and  are  described  in  detail  in  Chapter  II.  For  all  bases,  4.7  percent  of  all 
missions  received  a  PPMC  from  their  PWFs,  and  3.7  percent  of  missions 
received  a  PPMC  from  their  MEFs.  The  PPMC  for  Beale  AFB,  Nellis  AFB,  and 
Whiteman  AFB  was  zero  because  there  was  little  negatively  impacting  weather 
phenomena  identified  in  the  forecasts  by  aviators,  thus  the  opportunity  to  have  a 
PPMC  was  extremely  low.  The  Kunsan  AB  PPMC  was  zero  because  aviators 
typically  made  changes  when  they  perceived  negative  mission  impacts  would 
occur  due  to  weather.  Again,  results  indicate  Osan  AB  had  the  highest  PPMC  at 
3.7  percent.  This  suggests  aviators  had  the  potential  to  achieve  their  primary 
mission  in  3.7  percent  of  all  missions  had  they  made  mission  plan  changes  when 
they  perceived  negative  mission  impacts  would  occur  due  to  MEFs. 

Overall,  PWFs  had  a  higher  PPMC  of  4.7  percent,  when  compared  to  the 
MEFs  at  3.7  percent,  while  the  PMC  was  the  same  for  PWFs  and  MEFs  (see 
figure  25).  This  indicates  the  importance  of  PWFs  to  mission  planning  and 
execution. 

C.  NPS  AVIATOR  SURVEY  RESULTS 

Of  the  36  NPS  pilots  and  navigators  surveyed,  15  provided  lengthy 
feedback  to  our  survey  (Figure  7).  These  survey  results  confirmed  the 
identification  from  our  ACC  and  PACAF  data  collection  of  surface  visibility,  cloud 
ceilings,  and  cloud  layers  as  the  three  weather  phenomena  resulting  in  the  most 
negative  mission  impacts,  and  therefore  the  three  of  most  interest  to  them  in 
forecasts.  The  survey  results  also  suggest  most  aviators  only  look  at  forecasts 
within  24  hours  of  takeoff  and  usually  only  within  six  hours,  unless  the  mission  is 
“other  than  normal”  training  such  as  a  “check  ride”  where  the  aviators  are 
upgrading  their  flying  qualifications.  Most  aviators  have  a  secondary  mission 
plan  when  stepping  to  their  aircraft  to  fly  and  also  have  the  flexibility  to  execute 
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the  secondary  plan  while  in  flight.  Thus,  they  will  deem  many  missions 
“successful”  even  though  the  primary  objective  was  not  accomplished. 

After  data  analyses  were  completed,  four  NPS  aviators  participated  in  a 
round-table  discussion  concerning  the  results.  All  aviators  that  had  worked  with 
weather  personnel  embedded  in  the  flying  squadron  felt  the  support  was  far 
superior  to  their  experiences  with  non-embedded  weather  personnel.  Aviators 
felt  embedded  weather  personnel  had  a  better  feel  for  the  environment  within 
which  they  operate  and  also  had  a  better  handle  on  mission-impacting  weather. 
Aviators  also  felt  the  weather  support  provided  by  embedded  CWT  personnel 
was  better  tailored  to  their  needs. 

When  asked  about  our  findings  which  indicate  PWFs  might  be  under 
utilized,  NPS  aviators  did  not  have  consistent  feedback  as  to  whether  or  not  they 
remembered  using  or  even  seeing  a  CWT-issued  PWF.  Feedback  indicated 
aviators  used  various  sources  for  planning  weather  purposes,  such  as  television 
and  internet  sources. 

Those  aviators  who  had  spent  at  least  two  years  as  instructors  in  Air 
Education  and  Training  Command  (AETC)  felt  they  spent  considerably  more  time 
planning  a  mission  in  AETC  than  they  did  when  flying  in  ACC  or  PACAF  units. 
Aviators  indicated  AETC  missions  are  driven  by  a  strict  syllabus  and  when 
planning  a  mission,  if  there  was  a  high  probability  of  not  achieving  the  directives 
in  the  syllabus,  they  would  cancel  instead  of  flying  anyway  and  trying  to 
accomplish  something  else.  This  confirms  our  assessment  that  the  definition  of  a 
“successful”  mission  varies  widely  among  flying  units.  It  also  suggests  that  a 
future  study  of  the  performance  and  operational  impacts  of  forecasts  issued  by 
AETC  CWTs  would  be  useful. 

D.  SUMMARY 

The  general  consensus  from  the  ACC  and  PACAF  CWTs  was  that  the 
data  collection  form  was  too  long  and  therefore  aircrews  were  reluctant  to 
complete  it.  The  form  was  also  not  structured  in  a  flow  consistent  with  aircrew 
thought  processes  after  they  had  just  completed  a  mission.  Several  data 
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collection  forms  were  found  where  aircrews  started  to  answer  mission  planning 
questions,  the  first  questions  on  the  form,  with  mission  execution  answers  and 
then  when  they  reached  the  mission  execution  questions,  realized  what  they  had 
done,  and  used  arrows  and  side  notes  to  explain.  A  proposed  re-structured  data 
collection  form  is  discussed  in  Chapter  IV. 

Since  the  focus  of  this  thesis  was  to  develop  a  prototype,  proof  of  concept 
system,  in  some  cases  there  was  not  enough  data  to  adequately  assess  results. 
However,  we  did  attempt  to  calculate  all  of  the  performance  and  impacts  metrics 
described  in  chapter  II  with  the  data  we  collected.  However,  for  six  of  the 
impacts  metrics,  we  felt  the  data  quality  and/or  quantity  was  not  sufficient  to 
justify  calculating  them. 

Of  the  107  missions  analyzed,  40  percent  were  from  PACAF  flying  units 
and  60  percent  from  ACC  units.  In  82  percent  of  the  missions,  aviators  deemed 
the  mission  successful,  which  we  interpreted  to  mean  that  if  the  aircrew  was  not 
able  to  accomplish  the  primary  mission,  then  an  alternate  plan  was  successfully 
executed.  In  the  seven  percent  of  unsuccessful  missions  due  to  weather 
phenomena,  surface  visibility  was  the  primary  cause.  The  primary  weather 
phenomena  that  caused  negative  impacts  to  missions  were  surface  visibility, 
cloud  layers,  and  cloud  ceilings.  Of  these  three,  the  forecasts  did  the  poorest  job 
at  predicting,  and/or  aviators  did  the  poorest  job  at  interpreting,  cloud  layers. 

Forecast  performance  metrics  indicated  the  mean  FACs  for  the  PWFs  and 
the  MEFs  were  69.2  percent  as  assessed  by  aviators,  which  is  slightly  higher 
than  the  mean  No  Go  MEF  FAC  in  previous  studies  where  weather  personnel 
assessed  the  accuracy  (e.g.,  Jarry  2005).  Results  indicated  a  mean  POD  of  65.5 
percent  for  both  the  PWF  and  MEF,  which  is  slightly  lower  than  previous  studies; 
for  example,  a  mean  No  Go  MEF  POD  of  67.2  percent  from  Jarry  (2005).  The 
mean  PWF  FAR  was  31.2  percent  and  the  mean  MEF  FAR  was  52.1  percent. 
Jarry  (2005)  found  a  mean  No  Go  MEF  FAR  at  34.4  percent  for  AMC  in  fiscal 
year  2004. 
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Operational  impacts  metrics  indicated  both  the  PWF  and  the  MEF  had  a 
PMC  of  10.2  percent.  This  may  not  be  so  important  in  a  training  environment, 
but  in  a  war-time  environment  where  the  aviators  must  accomplish  their  primary 
mission,  this  would  most  likely  be  important.  Additionally,  PWFs  had  a  PPMC  to 
all  missions  analyzed  of  4.7  percent  while  the  MEF  PPMC  was  3.7  percent.  This 
suggests  more  emphasis  should  be  placed  on  PWFs,  especially  when  several  of 
the  NPS  aviators  indicated  they  often  times  looked  for  planning  weather  from 
sources  other  than  their  CWTs  because  they  were  not  aware  their  CWTs  issued 
PWFs. 

Many  of  the  values  we  obtained  in  our  calculation  of  forecast  performance 
and  operational  impacts  metrics  were  similar  to  those  obtained  in  prior  studies 
using  much  larger  data  sets  (e.g.,  Hinz  2004,  Jarry  2005).  This  suggests  that, 
although  we  collected  a  limited  amount  of  data,  and  collected  it  from  aviators,  not 
CWT  personnel,  the  data  quantity  and  quality  were  still  sufficient  to  allow  us 
calculate  useful  metrics. 

The  online  data  collection,  analysis  and  reporting  part  of  our  system  was 
not  sufficiently  tested  by  the  aviators  to  assess  its  effectiveness  from  the  aviator’s 
perspective.  However,  we  suspect  that  web-based  collection  by  aviators  may  be 
more  of  a  challenge  than  completing  paper  copies  because  of  the  need  to  have  a 
computer  available  to  every  aviator  at  the  time  of  their  debriefs.  Collecting  data 
in  locations  where  weather  personnel  are  not  fully  integrated  into  the  flying 
squadrons  was  a  challenge  due  to  the  interface  needed,  and  NPS  aviators 
verified  the  strain  on  CWT-aviator  coordination  when  weather  personnel  are  not 
embedded  in  the  flying  squadrons.  However,  CWT  manning  plays  a  huge  role  in 
whether  or  not  full  integration  is  accomplished  and  we  saw  a  reflection  of  low 
CWT  manning  in  the  lower  amounts  of  data  collected. 

There  were  sharp  contrasts  in  the  amount  of  data  collected  from  ACC  and 
PACAF  units,  primarily  due  to  differences  in  the  occurrence  of  negatively 
impacting  weather,  which  was  not  as  common  for  the  ACC  units  during  our  data 
collection  period. 
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IV.  SUMMARY,  DISCUSSIONS,  AND  RECOMMENDATIONS 


A.  SUMMARY  OF  PROCEDURES  AND  RESULTS 

Many  new  and  existing  challenges  face  AFW  personnel  in  today’s 
resource-limited  and  changing  environment.  Accurate  weather  forecasts  are 
imperative  in  aviation  planning  and  execution  of  combat  operations.  To  ensure 
forecast  accuracy  and  operational  relevancy,  quantitative  analyses  of  forecast 
performance  and  operational  impacts  are  essential,  but  should  be  done  in  the 
most  efficient  and  effective  manner  possible.  For  this  reason,  an  existing  web- 
based,  near-real  time  system  for  collecting  and  analyzing  weather  and  aviation 
data  was  adapted  for  use  by  AFW,  ACC,  and  PACAF.  Consistent  with  Figure  9, 
the  data  came  from  planning  and  execution  forecasts  as  well  as  observed 
weather  phenomena  (left  side  of  Figure  9),  and  from  aviation  planning  and 
execution  feedback  (right  side  of  Figure  9),.  Although  the  data  was  limited,  we 
were  able  to  calculate  metrics  which  appear  to  provide  useful  information  about 
overall  forecast  performance  and  operational  impacts.  Due  to  lack  of  negatively 
impacting  weather  phenomena  at  the  ACC  bases,  much  of  the  data  on  negatively 
impacting  weather  came  from  PACAF  bases.  However,  our  process  and  results 
can  be  applied  to  all  AFW  CWTs. 

1.  Goals  Accomplished 

The  first  goal  we  set  out  to  accomplish  was  to  adapt  a  system  already  in 
use  by  the  U.S.  Navy  for  collecting  and  analyzing  data  in  near-real  time.  This 
was  much  more  challenging  than  anticipated  because  our  data  collection  and 
analyses  were  done  by  aviators  and  not  weather  personnel,  which  resulted  in  the 
rewrite  of  much  of  the  computer  code. 

Our  second  goal  was  to  implement  and  test  the  system  with  seven  ACC 
and  PACAF  CWTs.  Six  of  the  seven  CWTs  were  able  to  participate  in  data 
collection.  But,  due  to  IT  challenges  and  time  constraints,  the  online  portion  of 
the  system  was  not  operational  until  much  of  the  data  had  already  been 
collected.  However,  we  did  receive  feedback  from  two  CWTs  indicating  it  would 
not  have  been  a  good  idea  to  have  aviators  input  the  data  directly  into  an  online 
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system  instead  of  filling  out  a  paper  copy  of  the  data  collection  form.  CWT 
personnel  directly  entering  data  into  the  online  form  is  a  better  option.  Of  the  six 
CWTs  that  provided  data,  three  felt  they  gained  valuable  feedback  on  ways  they 
could  improve  aviator  support  through  the  process  of  coordinating  the  data 
collection  form  with  the  aviators. 

Our  third  goal  was  to  use  the  results  from  system  implementation  to 
provide  AFW,  ACC,  and  PACAF  operations  advice  on  how  to  alter  procedures 
and  allocation  of  resources  to  better  accomplish  goals.  We  ended  up  calculating 
five  quantitative  metrics  of  forecast  performance  and  15  quantitative  metrics  of 
operational  impacts. 

2.  Results 

A  total  of  107  missions  were  analyzed;  82  percent  deemed  successful, 
seven  percent  deem  unsuccessful,  and  11  percent  not  assessed  by  aviators. 
Aviators  inferred  negative  mission  impacts  from  36  percent  of  the  PWFs,  but  only 
made  mission  plan  changes  to  21  percent  of  the  missions  in  response  to  the 
PWFs.  Aviators  inferred  negative  mission  impacts  from  39  percent  of  the  MEFs, 
but  again,  only  made  mission  plan  changes  to  21  percent  of  the  missions  in 
response  to  these  MEFs.  However,  aviators  encountered  negatively  impacting 
weather  phenomena  in  36  percent  of  their  missions. 

Forecast  performance  metrics  indicated  a  mean  FAC  of  69.2  percent  for 
both  PWFs  and  MEFs.  The  mean  POD  for  both  PWFs  and  MEFs  was  65.5 
percent.  Results,  for  PWFs  and  MEFs,  deviated  when  the  FAR  was  calculated. 
The  mean  FAR  for  PWFs  was  31.2  percent  while  the  mean  MEF  FAR  was  52.1 
percent.  This  gives  PWFs  a  slight  edge  over  MEFs,  which  is  important  to  note 
since  CWTs  spend  significantly  more  time  formulating  MEFs  than  PWFs. 

Operational  impacts  were  measured  by  relating  MEF  performance  metrics 
to  operational  performance  metrics.  We  determined  that  surface  visibility,  cloud 
ceilings,  and  cloud  layers  were  the  weather  phenomena  that  most  adversely 
affected  missions.  Surface  visibility  was  cited  as  the  cause  in  71  percent  of  the 
missions  deemed  unsuccessful  due  to  weather,  which  represents  five  percent  of 
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all  missions.  Aircrews  made  four  types  of  mission  plan  changes,  but  only 
changed  the  mission  type  (e.g.  low-level,  refueling,  and  air-to-air)  due  to  PWFs, 
never  MEFs.  This  indicates  PWFs  may  need  to  be  given  more  time  and  attention 
by  CWTs  and  aircrews. 

PWFs  and  MEFs  both  provided  a  PMC  to  10.2  percent  of  all  successful 
missions.  As  explained  in  Chapter  II,  a  forecast  has  a  PMC  when  the  aviator 
deems  negative  mission  impacts  will  result  from  weather  phenomena,  makes 
mission  plan  changes  based  on  the  forecast,  and  subsequently  achieves  their 
primary  mission.  Also  explained  in  Chapter  II  is  a  PPMC.  A  PPMC  occurs  when 
an  accurate  forecast  implies  negative  mission  impacts,  aviators  make  no  mission 
plan  changes,  and  then  the  primary  mission  is  not  accomplished.  The  PWFs  had 
a  PPMC  in  4.7  percent  of  all  missions,  while  the  MEFs  had  a  PPMC  of  3.7 
percent.  Once  again,  this  suggests  more  emphasis  needs  to  be  placed  on 
PWFs. 

The  NPS  aviator  survey  validated  findings  of  the  top  three  weather 
phenomena  needing  further  research  and  training.  The  high  percentage  of 
successful  missions  was  justified  by  NPS  aviators  who  explained  that  missions 
they  had  flown  while  in  ACC  and  PACAF  were  deemed  successful  if  they 
accomplished  something  useful,  even  if  they  did  not  accomplish  their  primary 
mission.  They  reported  that  it  was  common  for  aircrews  to  have  a  secondary 
mission  plan  and  the  flexibility  to  execute,  if  needed.  However,  aviators  who  had 
flown  in  both  AETC  and  either  ACC  or  PACAF  described  a  notable  difference 
when  flying  in  AETC,  versus  ACC  or  PACAF.  At  ETC,  mission  plans  are  closely 
tied  to  a  syllabus,  with  no  secondary  plan,  so  success  is  determined  by  following 
the  syllabus.  NPS  aviators  also  provided  valuable  feedback  regarding  the  CWT 
products.  Most  of  the  NPS  aviators  were  not  confident  they  knew  how,  when, 
and  where  to  access  all  of  the  CWT  products,  especially  PWFs.  Some  were 
unaware  that  the  CWTs  at  the  flying  bases  they  had  been  assigned  issued  formal 
PWFs,  others  received  planning  weather  by  phoning  the  CWT,  and  yet  others 
looked  up  planning  weather  via  other  sources  on  the  web. 
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3.  Meeting  AFW  Challenges  and  Vision 

Implementing  an  online  system  such  as  the  one  developed  and  tested  in 
this  research  project,  will  help  improve  mission  efficiency  and  effectiveness  of 
CWTs.  The  results  of  this  project  will  help  enable  CWTs  to  compensate  for  not 
having  sufficient  personnel  to  fully  integrate  themselves  into  all  the  flying 
squadrons  they  support. 

Step  11  of  the  MEFP  as  outlined  in  chapter  four  of  AFMAN  15-129  directs 
the  MEF  be  a  product  for  which  CWTs  receive  continuous  feedback  from 
aviators.  The  system  in  this  research  project  provides  continuous  aviator 
feedback.  The  first  task  in  step  12  of  the  MEFP  as  outlined  in  chapter  four  of 
AFMAN  15-129  calls  for  the  implementation  of  systematic  procedures  to  analyze 
and  measure  accuracy/relevancy  of  products  provided  to  customers.  Step  two 
directs  operator  debriefs,  preferably  face-to-face,  but  when  this  is  not  possible  it 
directs  CWTs  to  employ  other  feedback  methods.  Step  five  directs  CWTs  to 
conduct  operations  verification  on  established  “Go/No  Go”  thresholds  and  step 
six  requires  the  development  and  application  of  metrics  for  process  improvement 
(AFMAN  15-129  2004).  Our  system  meets  or  could  readily  be  adapted  to  meet, 
all  of  these  directives. 

Guidance  in  AFI  15-114  deems  operational  verification  of  the  MEF  as  the 
“single  most  important  mission-oriented,  operational  effectiveness  assessment 
requirement  for  CWTs.”  It  also  calls  for  a  system  that  “automates  all  metrics, 
from  data  collection,  aggregation,  to  data  quality  control.  The  end-state  will  be 
an  automated  web-based  system  with  the  capability  to  provide  ad  hoc  analyses 
and  reports  (assessments)  for  all  levels  of  AFW  support”  (AFI  15-114).  Our 
system  meets  the  AFI  guidance  above. 

The  AFAA  report  released  September  2005  found  that  Air  Force 
personnel  did  not  consistently  and  completely  obtain  customer  feedback  and 
identify  improvement  opportunities,  and  recommended  that  AFW  establish 
standardized  procedures  for  obtaining  mission  operator  feedback  on  weather 
forecasts  (AFAA  2005).  Our  system  is  a  solution  to  this  finding. 
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The  AFWSPV  set  the  course  for  transformation  starting  with  fiscal  year 
2008.  The  process  we  are  proposing  in  this  thesis  takes  steps,  however  small, 
toward  three  of  the  eight  vectors.  It  creates  automated  decision  tools  which  can 
further  be  exploited  (vector  6),  it  helps  determine  where  weather  information  is 
integrated  into  decision  cycles  at  the  base  level  and  below  (vector  7),  and 
although  not  directly,  it  enables  similar  feedback  results  that  could  be 
accomplished  by  embedding  weather  personnel  into  flying  squadrons  (vector  8) 
(AFWSPV  2004). 

B.  RECOMMENDATIONS 

The  task  list  for  CWTs  is  lengthy  and  manpower  challenges  make 
achieving  many  of  the  tasks  unrealistic.  Implementation  of  the  prototype  system 
we  have  adapted  for  usage  at  Air  Force  bases  will  enable  AFW  to  alleviate  some 
of  the  challenges  facing  CWTs.  Specifically,  providing  this  prototype  as  an 
appendix  to  AFI  15-114  will  provide  CWTs  an  example  of  an  automated  system 
that  is  already  working.  It  also  eliminates  CWTs  from  having  to  train  personnel 
on  metrics  calculations. 

Although  missions  vary  drastically  from  fighter  aircraft  to  bombers,  to 
reconnaissance  and  surveillance,  etc.,  generic  data  collection  forms  can  be 
drafted  for  each  platform.  Our  results  indicate  our  data  collection  form  was  too 
long  and  the  order  in  which  questions  were  asked  needed  to  be  changed. 
Questions  pertaining  to  the  actual  mission  need  to  be  asked  first  because 
aviators  complete  the  forms  immediately  after  they  have  flown  their  mission. 
Figure  27  is  a  revised  generic  data  collection  form  for  fighter  aircraft.  We 
recommend  similar  forms  be  created  for  differing  platforms.  Once  these  data 
collection  forms  are  produced,  they  can  be  used  in  our  prototype  system  for  data 
collection. 

When  we  started  coordination  with  the  original  seven  CWTs,  it  became 
apparent  there  is  a  need  in  the  AFW  community  to  formalize  training  on  getting  to 
know  your  customer.  We  recommend  that  a  formal  training  process  and 
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documents  be  developed  and  implemented,  and  that  CWT  personnel  be  given 
enough  overlap  with  those  they  replace  to  allow  for  ample  orientation  to  the  flying 
squadrons  they  will  be  supporting. 

The  NPS  aviators  could  not  praise  embedded  weather  support  in  the  flying 
squadrons  enough.  Aviators  indicated  the  value  added  when  weather  personnel 
were  available  in-person  at  the  flying  squadron  was  immeasurable.  Further,  they 
felt  that  once  a  weather  person  experienced  the  environment  in  which  aviators 
operate,  they  were  able  to  understand  weather  requirements  better  and 
established  credibility  with  the  aviators.  We  recommend  AFW  leadership 
reinforce  flying  squadron  integration  and  CWT  manning.  Feedback  from  CWTs 
during  our  coordination  and  data  collection  processes  indicated  that  air-to-ground 
platforms  are  the  most  “weather  needy”  and  emphasis  should  be  placed  on 
manning  CWTs  accordingly.  Further,  we  recommend  an  operational 
effectiveness  study  be  conducted  to  compare  the  quantitative  impact  of  having 
embedded  weather  personnel  in  the  flying  squadrons  to  not  having  embedded 
weather  personnel. 

Several  of  our  results  suggest  that  aviators  are  not  familiar  with  many  of 
the  terms  AFW  personnel  use.  The  feedback  we  received  also  suggests  AFW 
personnel  need  to  educate  aviators  better  on  the  types  of  products  available  and 
when  and  where  aviators  can  find  these  products. 

Results  indicate  AFW  forecasts  have  the  ability  to  provide  positive  mission 
impacts  when  aviators  follow  forecasts  and  make  mission  plan  changes 
accordingly.  It  appears  PWFs  can  make  a  difference  in  whether  or  not  aviators 
achieve  their  primary  mission.  So  we  recommend  that  aviators  better  incorporate 
weather  products  earlier  in  their  planning  process,  at  least  12-24  hours  prior  to 
takeoff . 

C.  FUTURE  WORK 

Work  already  in  progress  at  the  NPS  is  adapting  this  web-based,  near-real 
time  system  for  collecting  and  analyzing  METOC  and  aviator  data  from  U.S. 
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Navy  aircraft  carriers.  The  same  system  being  adapted  across  U.S.  military 
service  lines  will  improve  joint  service  cooperation. 

Feedback  from  CWTs  and  NPS  aviators  stressed  the  large  number  of 
different  agencies  wanting  feedback  from  aviators  after  mission  execution.  Thus, 
it  would  be  useful  to  incorporate  weather  data  collection  forms  into  other  existing 
pilot  debrief  systems.  Perhaps  airfield  flight  and  weather  could  combine  their 
data  collection  efforts  into  one  database.  Semi-automated  feedback  needs  to 
continue,  but  will  not  be  successful  unless  it  is  tied  into  an  automated  system  that 
is  already  established.  Evidently,  such  a  system  is  already  setup  at  Whiteman 
AFB  called  Patriot  Excalibur.  Unfortunately,  the  system  is  not  accessible  from  a 
non-“.mil”  website  therefore  limiting  usage. 

Adapting  our  prototype  system  to  combat  operations,  such  as  Operation 
Iraqi  Freedom,  would  provide  valuable  “war-time”  information.  This  would  need 
to  be  accomplished  over  a  secure  network  which  would  involve  a  lot  more  time 
due  to  limited  availability  to  communicate  with  the  data  collectors  in  theater,  but 
would  be  extremely  valuable. 

The  biggest  challenge  we  faced  was  getting  data.  The  fast-paced  aviator 
environment  proved  to  be  a  challenging  one  to  tap  into.  Unless  aviators  can  be 
provided  qualitative  and  quantitative  justification  as  to  why  it  is  advantageous  for 
them  to  provide  feedback,  it  will  continue  to  be  the  most  challenging  part  of 
operational  verification  for  the  AFW  community. 
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APPENDIX 


FIGURES 


Schematic  Timelines  Linking 
Aviation  Operations  and  Weather  Support 


Figure  1.  Schematic  timeline  linking  aviation  operations  to  weather  support. 

Timeline  begins  on  left  with  long  range  planning  that  occurs  three  months 
prior  to  mission  takeoff  to  the  actual  mission  execution  on  the  right. 
Linkage  provides  framework  for  determining  data  collection  opportunities. 
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Operations  Process  Data  to  be  Collected 


Figure  2.  Flow  chart  for  planning,  executing,  and  debriefing  air  operations  (blue 
boxes),  and  corresponding  data  collection  needed  for  analyses  of  the 
performance  and  operational  impacts  of  weather  forecasts  (yellow  boxes). 
The  operations  flow  chart  indicates  the  types  of  weather  products  aviators 
use  at  different  stages  in  their  planning  and  execution. 
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Mission  Successful 


Figure  3.  Flow  chart  depicting  all  avenues  mission  planning  and  execution  can  take 
from  receipt  of  a  weather  forecast  to  completion  of  a  successful  mission. 
Flow  chart  traces  back  the  forecast-related  steps  that  led  to  a  successful 
mission.  Steps  are  those  that  involve  the  planning  weather  forecast 
(PWF,  bottom  row),  mission  changes  made  in  response  to  PWF,  second 
row  up),  mission  execution  forecast  (MEF,  third  row  up),  mission  changes 
made  in  response  to  MEF  (fourth  row  up),  and  actual  weather  experienced 
during  the  mission  (fifth  row  up). 
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Figure  4.  Flow  chart  depicting  all  avenues  mission  planning  and  execution  can  take 
from  receipt  of  a  weather  forecast  to  completion  of  an  unsuccessful 
mission.  Flow  chart  traces  back  the  forecast-related  steps  that  led  to  an 
unsuccessful  mission.  Steps  are  those  that  involve  the  planning  weather 
forecast  (PWF,  bottom  row),  mission  changes  made  in  response  to  PWF, 
second  row  up),  mission  execution  forecast  (MEF,  third  row  up),  mission 
changes  made  in  response  to  MEF  (fourth  row  up),  and  actual  weather 
experienced  during  the  mission  (fifth  row  up). 
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Generic  Data  Collection  Form 


Mission  Identification 


Mission  #/Call  Sign: 

MO/DA/YR _ Take-off  Time  (Zulu). 

PLANNING 


1.  How  many  hours  prior  to  your  flight  did  you  first  check  the  Combat  Weather 
Team  (CWT)  weather? 

48,  36,  24,  12,  6,  0 

2.  How  many  hours  prior  to  your  flight  would  you  have  checked  the  CWT  weather 
if  it  were  available? 

72,  60,48,  24,12,0 

3.  What  negative  mission  impacts  were  forecasted  in  the  planning  weather? 
Won’t  see  target 

Won’t  be  able  to  lock-on  target 

Won’t  be  able  to  do  air-to-air  training 

Won’t  be  able  to  take-off  or  land 

Won’t  be  able  to  refuel 

Won’t  be  able  to  do  instrumentation  training 

Won’t  be  able  to  do  touch  and  gos 

Won’t  be  able  to  do  low-level  training 

None 

4.  What  weather  phenomena  led  to  these  forecasted  negative  impacts? 

Cloud  Ceiling 

Cloud  Layers 

Thunderstorms 

Rain 

Visibility  at  Surface 

Visibility  Aloft 

Turbulence 

Icing 

X-wind 

None 

5.  What  changes  in  mission  plans  were  made  due  to  planning  weather 
forecasting  negative  impacts? 

Aircrew 

Weapon 

Schedule  (i.e.  time  of  mission) 
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EXECUTION 


6.  How  many  hours  prior  to  your  flight  did  you  check  the  Mission  Execution 
Forecast  (MEF)? 

6.  5,  4,  3,  2,  1,0 

7.  What  negative  mission  impacts  were  forecasted  in  MEF? 

Won’t  see  target 

Won’t  be  able  to  lock-on  target 

Won’t  be  able  to  do  air-to-air  training 

Won’t  be  able  to  take-off  or  land 

Won’t  be  able  to  refuel 

Won’t  be  able  to  do  instrumentation  training 

Won’t  be  able  to  do  touch  and  gos 

Won’t  be  able  to  do  low-level  training 

None 

8.  What  weather  phenomena  led  to  these  MEF  negative  impacts? 

Cloud  Ceiling 

Cloud  Layers 

Thunderstorms 

Rain 

Visibility  at  Surface 

Visibility  Aloft 

Turbulence 

Icing 

X-wind 

None 

9.  What  additional  changes  in  mission  plans  were  made  due  to  MEF  negative 
impacts? 

Aircrew 

Weapon 

Schedule  (i.e.  time  of  mission) 

Type  (i.e.  high  alt,  instrumentation,  strike,  low-level) 

None 

10.  What  negative  impacts  due  to  weather  actually  occurred  during  mission? 
Couldn’t  take-off 

Couldn’t  see  target 
Couldn’t  refuel 

Couldn’t  perform  low-level  training 
Couldn’t  perform  air-to-air  training 
Couldn’t  perform  instrumentation  flight 
None 
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11.  What  weather  phenomena  led  to  these  negative  impacts  during  your 
mission? 

Cloud  Ceiling 
Cloud  Layers 
Thunderstorms 
Rain 

Visibility  at  Surface 

Visibility  Aloft 

Turbulence 

Icing 

X-wind 

None 

12.  How  are  you  using  Target  Acquisition  Weapons  Software  (TAWS)? 

Not  using  (skip  to  question  #21) 

Mission  planning 

Execution 

Both 

13.  Does  TAWS  have  your  weapon  sensor  modeled? 

Yes 

No 

14.  What  are  you  using  TAWS  for? 

IR 

TV 

Laser 

NVG  predictions 

15.  Did  you  change  your  weapons  load  or  tactics  based  on  TAWS  predictions? 
Yes 

No 

16.  How  were  the  TAWS  predictions? 

Pessimistic 

Optimistic 

Accurate 

17.  Did  you  download  weather  from  AFWA  or  get  a  TAWS  file  from  your  CWT? 
AFWA 

CWT 

18.  Was  the  TAWS  weather  accurate? 

Yes 

No 
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19.  Would  you  use  TAWS  again? 

Yes,  but  it  could  be  improved  by _ (insert  your  suggestions) 

No 


20.  Rate  the  prediction  software  in  the  areas  below  using  the  following  scale: 
1-not  useful,  2-somewhat  useful  but  not  required,  3-useful,  4-mission  essential 


Pre-mission: 

a.  Route  selection  12  3 

b.  Target  viewing  azimuth  12  3 

c.  Target  area  tactics  12  3 

Inflight: 

a.  Low  level  navigation  12  3 

b.  Target  area  tactics  12  3 

c.  Target  identification  12  3 

d.  Threat  avoidance  12  3 

e.  Sensor  cross-check  techniques  12  3 


4 

4 

4 


4 

4 

4 

4 

4 


21.  What  did  you  intend  to  accomplish  on  your  mission? 

Low-level  training 

High-altitude  training 

Instrumentation  training 

Air-to-Air  Combat  training 

Air-to-Air  Refueling 

Strike  Target  on  Ground 


22.  What  did  you  actually  accomplish  on  your  mission? 

Low-level  training 

High-altitude  training 

Instrumentation  training 

Air-to-Air  Combat  training 

Air-to-Air  Refueling 

Strike  Target  on  Ground 

Nothing 


23. 


a. 


b. 


c. 


What  was  your  overall  impression  of  the  actual  mission? 


□ 

Mission  successful 

□ 

Mission  unsuccessful 

□ 

Changed  mission  due  to 

□ 

No  change  in  mission  due  to 

□ 

planning  weather 

□ 

planning  weather 

Changed  mission  due  to  MEF 

No  mission  change  due  to  MEF 

d. 


□ 

□ 

□ 


No  negative  weather  impacts  forecasted 
Negative  weather  impacts  forecasted  accurately 
Negative  weather  impacts  forecasted  inaccurately 


e. 


No  WX  impact  to  mission 


□ 
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Negative  WX  impact  to  mission 


Figure  5.  Generic  data  collection  form  used  to  gain  feedback  on  planning,  execution 
and  TAWS  data  from  aviators  after  mission  execution. 


IRTSS  Questionnaire 

Nellis  AFB  is  the  lead  ACC  unit  for  software  that  creates  Sensor 
Prediction  Products  (SPP).  Infrared  Target  Scene  Simulation  (IRTSS)  is 
the  latest  and  greatest  SPP  tool.  Thanks  for  your  time  and  for  helping 
make  the  program  better. 

Which  sensor[s]  did  you  employ  on  your  mission? 

Was  the  IRTSS  prediction  helpful  in  your  mission  planning? 

Was  the  IRTSS  prediction  a  realistic  representation  of  what  you  saw  on 
your  cockpit  display? 

Did  IRTSS  represent  'feeder  features'  (roads,  rivers,  etc)  into  the  target 
area  correctly? 

Did  IRTSS  represent  the  target  area  correctly? 

Did  the  IRTSS  prediction  influence  you  to  adjust/alter  tactical  decisions 
before  mission  execution? 

What  is  one  enhancement  that  if  made  to  IRTSS  would  make  the  product 
more  useful  to  you? 

Figure  6.  Infrared  target  scene  simulation  (IRTSS)  data  collection  form  used  to  gain 
feedback  from  aviators  after  mission  execution. 
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Naval  Postgraduate  School  Aviation  Survey  Form 

1 .  Please  provide  a  brief  history  of  your  flying  experience  to  include 
airframes  flown  and  number  of  hours  in  that  airframe. 

2.  In  general,  how  many  days  prior  to  a  mission  did  you  first  look  at  the 
weather?  How  did  you  receive  this  weather?  (internet,  base  weather 
station,  weather  channel)  What  were  you  looking  for?  (general  sky 
coverage,  thunderstorms,  precipitation,  fog,  solar/lunar  data) 

3.  In  general,  how  flexible  were  you  at  making  changes  to  the  mission 
plan?  Please  elaborate  on  how  maintenance  and  perhaps  airfield 
operations  play  into  this.  Discuss  weapons,  targets,  scheduling 
ranges,  refueling,  scheduling  low-level  routes,  different  category 
pilots. 

4.  In  your  opinion,  what  are  the  most  important  parts  of  the  MEF? 
(Mission  Execution  Forecast  issues  by  the  Combat  Weather  Team  at 
your  installation)  Which  weather  phenomena  (icing,  turbulence, 
visibility  at  surface,  etc.)  in  your  opinion,  caused  the  most  mission 
planning  changes  on  your  part? 

5.  From  your  flying  experiences,  what  does  Air  Force  Weather  (AFW)  do 
well  and  what  could  AFW  stand  to  put  time  and  effort  into? 

6.  In  general,  how  much  faith  have  you  had  in  the  weather  forecasts 
you’ve  received?  Did  you  find  the  quality  of  the  weather  support  you 
received  changed  based  on  weather  flight  leadership?  Person  to 
person? 

7.  When  you  first  get  to  a  new  location,  do  you  tend  to  pay  more 
attention  to  the  weather  forecast  versus  having  been  at  a  location  for 
a  year  where  you’ve  experienced  all  four  seasons  and  in  the  back  of 
your  mind  can  picture  what  you’ll  see  based  on  a  satellite  photo? 
Seems  like  common  sense,  but  just  want  to  confirm. 

8.  How  well  is  weather  integrated  into  operations?  Long  range  climate 
forecasts?  Planning  weather?  MEF? 

Figure  7.  Naval  Postgraduate  School  aviator  survey  form.  Information  from  this 
survey  was  used  to  validate  and  interpret  results  of  ACC  and  PACAF  data 
analyses. 
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Ave  #  Msns/day 

Ave  length  of  Msn 
Mission  Type 

A-10 

Thunderbolt  II 

B-1B 

Lancer 

B-2 

Spirit 

F-16 

Fighting  Falcon 

U-2 

UAV  (Unmanned 
Aerial  Vehicle) 

Beale  AFB,  CA 

5  Ave  Msns/day 

5  Hrs/Msn 

S&R 

Creech  AFB,  NV 

9  Ave  Msns/day 

8  Hrs/Msn 

S&R 

Dyess  AFB,  TX 

10  Ave  Msns/day 

6  Hrs/Msn 

B 

Kunsan  AB,  ROK 

1 8  Ave  Msns/day 
1.8  Hrs/Msn 

AA,  AG 

Nellis  AFB,  NV 

8  Ave  Msns/day 

2  Hrs/Msn 

AA,  AG,  CAS 

8  Ave  Msns/day 

2  Hrs/Msn 

AA,  AG 

Osan  AB,  ROK 

10  Ave  Msns/day 

3  Hrs/Msn 

AA,  AG,  CAS 

1 5  Ave  Msns/day 
1.5  Hrs/Msn 

AA,  AG 

Whiteman  AFB,  MO 

9  Msns/day 

4  Hrs/Msn 

B 

S&R=Surveillance  and  Reconnaissance  B=Bombing 

AA=Air-to-Air  CAS=Close  Air  Support 

AG=Air-to-G  round 


Figure  8.  Summary  of  missions  flown  during  data  collection  timeframe,  categorized 
by  air  base  (rows)  and  air  frame  (columns).  Information  on  missions 
describes:  average  number  of  missions  flown  per  day,  average  length  of 
mission  (hours),  and  mission  type  (see  key  at  bottom  of  table). 
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Figure  9.  Schematic  data  and  methods  flow  chart  showing  the  data  collected  (top 
row),  performance  analysis  (middle  row),  and  assessment  of  operational 
impacts  (bottom  row). 
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Data  Analyses  Calculations 

(Based  on  Kunsan,  Nellis,  and  Osan  data  collection  form) 

A.  Number  of  total  missions,  successful  missions  and  unsuccessful  missions 
analyzed 

1.  Count  the  number  of  total  missions  entered  into  the  database,  the 
number  of  missions  deemed  successful  in  #23a,  and  the  number  of 
missions  deemed  unsuccessful  in  #23a 

2.  Display  results  in  numeric  form  showing  total  missions,  successful 
missions,  and  unsuccessful  missions 

B.  Percentage  of  missions  where  aviator  would  have  checked  the  planning 
weather  earlier  had  it  been  available. 

1 .  If  #2  is  greater  than  #1 ,  then  divide  the  number  of  times  this  exists 
by  the  total  number  of  missions  analyzed  and  multiply  by  100 

2.  Display  results  in  a  bar  graph  with  time  (in  hours)  increments  (72, 
60,  48,  24,  12,  0)  on  the  x-axis  and  frequency  of  occurrence  on  the  y- 
axis 

C.  Percentage  of  missions  deemed  unsuccessful  and  the  aviators  would 
have  checked  the  planning  weather  earlier  had  it  been  available. 

1.  If  #23a  shows  mission  unsuccessful  and  #2  is  greater  than  #1,  then 
divide  the  number  of  times  this  exists  by  the  total  number  of  missions 
analyzed  and  multiply  by  100 

2.  Display  results  in  a  pie  chart  with  the  total  number  unsuccessful 
missions  as  the  total  pie  and  the  missions  where  aviators  would  have 
checked  the  planning  weather  earlier  had  it  been  available  as  a  piece 
of  the  pie 

D.  Percentage  of  missions  considered  successful  even  though  aviators  did 
not  actually  accomplish  what  they  intended  to. 

1 .  If  #23a  shows  mission  successful  and  #21  and  #22  do  not  have  the 
same  answers,  then  divide  the  number  of  times  this  exists  by  the  total 
number  of  missions  analyzed  and  multiply  by  100 

2.  Display  results  as  a  pie  chart  with  total  number  of  successful 
missions  as  the  total  pie  and  the  missions  where  aviators  didn’t 
accomplish  what  they  intended  to  as  a  piece  of  the  pie 
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E.  Percentage  of  missions  that  were  successful  even  though  negative 
weather  impacts  were  forecasted  in  MEF,  but  aviators  did  not  make  any 
mission  changes  due  to  the  MEF. 

1.  If  #23a  shows  mission  successful  and  #7  shows  anything  but 
“None”  and  either  #9  shows  “None”  or  #23c  shows  “No  mission  change 
due  to  MEF”,  then  divide  the  number  of  times  this  exists  by  the  total 
number  of  successful  missions  and  multiply  by  100 

2.  Display  results  as  a  pie  chart  with  the  total  number  of  successful 
missions  being  the  total  pie  and  the  piece  of  the  pie  representing 
missions  where  negative  weather  impacts  were  forecasted  in  the  MEF, 
but  no  mission  plans  were  changed 

F.  Correlation  between  weather  phenomena  and  negative  mission  impacts 

1.  Match  responses  to  #3  and  #7  to  the  corresponding  responses 
other  than  “None”  for  #4  and  #8. 

2.  Display  results  as  a  chart  with  the  mission  impacts  on  one  axis, 
weather  phenomena  on  the  other  axis,  and  frequency  of  occurrence  in 
the  corresponding  square 

G.  Percentage  of  missions  the  aircrew  checked  the  MEF  6,  5,  4,  3,  2,  1,  0 
hours  prior  to  takeoff 

1 .  Count  each  response  to  #6  and  divide  it  by  the  total  number  of 
missions 

2.  Display  results  as  a  bar  graph  with  the  hours  on  the  x-axis  and 
percent  of  missions  on  the  y-axis 

H.  Percentage  of  missions  successful  where  there  were  changes  in  mission 
plans  due  to  planning  weather  and  aviators  actually  accomplished  what  they 
intended  to  (Positive  Mission  Contribution  due  to  PWF) 

1.  If  #23a  shows  “mission  successful”  and  #5  shows  anything  other 
than  “None”  and  #22  has  at  least  the  same  options  as  #21  and  #23b 
shows  “Changed  mission  due  to  planning  weather”,  divide  the  number 
of  times  this  exists  by  the  total  number  of  missions  analyzed  and 
multiply  by  100 

2.  Display  results  as  a  pie  chart  with  letter  I,  where  the  whole  pie  is  the 
total  number  of  successful  missions  analyzed  and  a  piece  of  the  pie  is 
the  PMC  due  to  PWF 
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I.  Percentage  of  missions  successful  where  there  were  changes  in  mission 
plans  due  to  MEF  and  aviators  actually  accomplished  what  they  intended  to 
(Positive  Mission  Contribution  due  to  MEF) 

1.  If  #23a  shows  “mission  successful”  and  #9  shows  anything  other 
than  “None”  and  #22  has  at  least  the  same  options  as  #21  and  #23c 
shows  “Changed  mission  due  to  MEF”,  divide  the  number  of  times  this 
exists  by  the  total  number  of  missions  analyzed  and  multiply  by  100 

2.  Display  results  as  pie  chart  with  letter  H,  where  the  whole  pie  is  the 
total  number  of  successful  missions  analyzed  and  a  piece  of  the  pie  is 
the  PMC  due  to  MEF 

J.  Correlation  between  weather  phenomena  forecasted  and  resulting  mission 
plan  changes 

1.  Most  frequent  responses  other  than  “None”  to  #4  and  #8  and  the 
corresponding  responses  to  #5  and  #9 

2.  Display  results  as  a  chart  with  the  mission  plan  changes  on  one 
axis,  weather  phenomena  on  the  other  axis,  and  frequency  of 
occurrence  in  the  corresponding  square 

K.  Weather  phenomena  in  MEF  resulting  in  the  most  unsuccessful  missions 

1 .  Identify  the  correlation  between  a  response  of  “Mission 
unsuccessful”  in  #23a  and  the  corresponding  answer  to  #8  other  than 
“None” 

2.  Display  each  weather  phenomena  on  x-axis  and  #  of  unsuccessful 
missions  on  y-axis 

L.  Percentage  of  missions  a  negative  impact  due  to  weather  actually 
occurred  when  MEF  did/didn’t  forecast  it 

1 .  If  #1 0  shows  anything  other  than  “None”  then  see  if  #7  shows  a 
corresponding  response  and  if  it  does,  then  it  “was  forecasted”;  if  it 
does  not,  it  “was  not  forecasted.”  Divide  both  the  “was  forecasted”  and 
the  “was  not  forecasted”  responses  for  each  impact  by  the  total 
occurring  and  multiply  by  100  to  get  percentages 

2.  Display  each  negative  mission  impact  on  x-axis  and  the  percentage 
of  missions  in  which  it  corresponded  to  the  #7  response  on  the  y-axis 

M.  Percentage  of  missions  a  particular  mission  plan  was  changed  due  to 
planning  weather  and  then  changed  again  due  to  MEF 

1 .  Responses  in  #5  other  than  “None”  that  correspond  to  the  same 
response  in  #9 

2.  Display  mission  plan  changes  on  x-axis  and  on  y-axis  show  three 
separate  percentages:  #  changed  due  to  planning  weather,  #  changed 
due  to  MEF,  #  plan  was  changed  both  due  to  planning  weather  and 
then  again  due  to  MEF 
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N.  Percentage  of  missions  where  mission  was  unsuccessful  and  operator  felt 
the  MEF  was  inaccurate. 

1 .  If  #23a  shows  “Mission  unsuccessful”  and  #8  responses  do  not 
match  #1 1  responses  and  #23d  shows  either  “No  negative  weather 
impacts  forecasted”  or  “Negative  weather  impacts  forecasted 
inaccurately”.  Divide  the  number  of  times  this  exists  by  the  number  of 
unsuccessful  missions  and  multiply  by  100 

2.  Display  results  numerically 

O.  The  most  common  weather  phenomena  actually  occurring  that  resulted  in 
mission  being  successful/unsuccessful. 

1 .  #11  responses  other  than  “None”  that  correspond  to  #23a  “mission 
successful/mission  unsuccessful” 

2.  Display  the  weather  phenomena  on  the  x-axis  and  number  of 
missions  on  the  y-axis.  Display  successful  missions  in  one  color  and 
unsuccessful  missions  in  another  color  on  the  bar  graph 

P.  Percentage  of  each  weather  phenomena  negatively  impacting  the 
mission,  but  was/wasn’t  forecasted  in  MEF 

1 .  If  response  to  #1 1  is  anything  other  than  “None”  and  it  wasn’t  in  #8 
responses 

2.  Display  the  weather  phenomena  on  the  x-axis  and  number  of  MEFs 
on  the  y-axis.  Display  both  when  they  correspond  and  when  they  do 
not  in  different  colors  on  the  bar  graph 

Q.  Percentage  of  missions  a  weather  phenomena  actually  occurred,  but 
wasn’t  in  the  MEF(planning  weather)  even  though  it  was  in  planning 
weather(MEF)  or  was  not  in  either  or  was  in  both 

1 .  If  response  to  #1 1  is  anything  other  than  “None”  and  it  wasn’t  in  #8 
(#4)  responses,  but  it  was  in  #4  (#8)  or  it  wasn’t  in  either  #8  or  #4  or  it 
was  in  both  #8  and  #4. 

2.  Display  the  weather  phenomena  on  the  x-axis  and  number  of  MEFs 
(planning  weather  forecasts)  on  the  y-axis.  Display  each  of  the  four 
situations  in  different  colors. 

Target  Acquisition  Weapon  Software  (TAWS) 

R.  Percentage  of  missions  that  used  TAWS 

1.  Count  the  number  of  responses  for  #13  that  responded  “Not  using” 
and  subtract  if  from  the  total  number  of  forms  submitted  and  then 
divide  this  number  by  the  total  number  of  missions  and  multiply  by  1 00. 

2.  Display  results  in  numeric  form. 
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S.  How  is  TAWS  being  used? 

1.  Responses  for  #13  other  than  “Not  using”  and  divide  each  response 
by  the  total  number  of  TAWS  users  and  multiply  by  100. 

2.  Display  results  in  a  bar  graph  with  “Mission  planning”,  “Execution”, 
and  “Both”  on  the  x-axis  and  percentage  of  TAWS  usage  on  the  y-axis 

T.  Percentage  of  time  TAWS  had  the  weapon  sensor  modeled  (didn’t  have 
the  weapon  sensor  modeled)  and  the  TAWS  predictions  were  accurate, 
pessimistic,  or  optimistic? 

1 .  Responses  to  #1 7  that  matched  a  “Yes”(“No”)  response  to  #1 4 

2.  Display  results  as  a  bar  graph  with  the  TAWS  predictions 
(pessimistic,  optimistic,  accurate)  on  the  x-axis  and  the  number  of 
times  these  responses  corresponded  to  the  weapon  sensor  being 
modeled  (“Yes”)  in  one  color  and  the  number  of  times  these  responses 
corresponded  to  the  weapon  sensor  not  being  modeled  (“No”)  in 
another  color 

U.  What  is  TAWS  being  used  for? 

1 .  Count  the  number  of  times  each  response  for  #1 5  is  marked  and 
divide  it  by  the  total  number  of  TAWS  users.  Multiply  this  by  1 00  to  get 
a  percentage. 

2.  Display  each  of  the  responses  to  #15  on  the  x-axis  and  the 
percentage  of  time  they  are  using  TAWS  for  it  on  the  y-axis 

V.  Percentage  of  missions  TAWS  weather  was  accurate/inaccurate  when  it 
came  from  the  CWT  versus  AFWA 

1 .  Count  the  number  of  times  each  response  for  #1 8  is  marked  and 
the  percentage  of  time  each  response  corresponded  to  a  #19  response 
of  “Yes”  or  “No” 

2.  Display  the  responses  to  #18  on  the  x-axis  and  the  percentage  of 
time  they  were  “Yes”  to  #1 9  in  one  color  and  the  percentage  of  time 
they  were  “No”  in  #19  in  another  color... the  percentages  are  on  the  y- 
axis 

W.  Percentage  of  TAWS  users  that  would  use  it  again  when  their  weapon 
sensor  was/wasn’t  modeled 

1 .  Count  the  number  of  times  each  response  for  #20  is  marked  and 
the  percentage  of  time  this  corresponded  to  either  answer  to  #14. 

2.  Display  each  of  the  responses  to  #20  on  the  x-axis  and  the 
percentage  of  time  their  weapon  sensor  was  modeled  in  one  color  and 
the  percentage  of  time  their  weapon  sensor  was  not  modeled  in 
another  color 
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X.  Percentage  of  time  the  weapons  load  or  tactics  were  changed  based  on 
TAWS  predictions 

1 .  Count  the  number  of  times  each  response  for  #1 6  is  marked  and 
divide  it  by  the  total  number  of  TAWS  users... multiply  this  by  100  to 
get  a  percentage. 

2.  Display  each  of  the  responses  to  #16  on  the  x-axis  and  the 
percentage  of  time  they  occurred  on  the  y-axis 

Y.  Usefulness  of  TAWS 

1 .  Count  the  number  of  times  each  response  for  #21  is  marked  and 
divide  it  by  the  total  number  of  TAWS  users... multiply  this  by  100  to 
get  a  percentage. 

2.  Display  each  of  the  responses  to  #21  on  the  x-axis  and  the 
corresponding  percentage  of  usefulness  (1-4)  on  the  y-axis  in  four 
different  colors... may  need  to  do  two  bar  graphs,  one  for  “Pre-mission” 
and  one  for  “In-flight” 

Figure  10.  Description  of  calculations  performed  to  analyze  ACC  and  PACAF  data, 
plus  instructions  for  displaying  the  results  of  the  analyses.  Based  on  data 
collected  from  Kunsan  AB,  Nellis  AFB,  and  Osan  AB. 
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copy  of  data 
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NPS 

Data  written  to  MYSQL 
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Data  analyzed  per  NPS 
metrics  procedures 


Data  entered  via 
HTML  &  PHP  webpage 


□  □  □  □ 


□  □  □  □ 


Output  reports  delivered  via 
webpage 

Reports  are  available  on 
demand  by  using  embedded 
PHP  to  calculate  metrics 

Reports  delivered  by  CWTs 
to  AFW  and  operations 
leadership 


ACC  &  PACAF  CWTs 


Figure  1 1 .  Flow  of  data,  from  collection  by  aviators,  to  analyses  at  Naval 
Postgraduate  School,  to  results  reported  to  leadership.  (Adapted  from 
Butler  2005) 
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2.  How  many  hours  prior  to  your  flight  would  you  have  checked  the  CWT  weather  if  it  were  available? 

r  72  re  0  r  48  r  24  r  12  r  6  r  0 

3.  What  negative  mission  impacts  were  forecasted  in  the  planning  weather? 

Select  all  that  apply 
r  Won't  see  target 
r  Won't  be  able  to  lock-on  target 
r  Won't  be  able  to  do  air-to-air  training 
r  Won't  be  able  to  take-off  or  land 
r  Won't  be  able  to  refuel 
r  Won't  be  able  to  do  instrumentation  training 
r  Won't  be  able  to  do  touch  and  go's 


Figure  12.  Online  data  collection  form  for  Osan  AB  available  online  at 
http://wx.met.nps.navv.mil/metrics/index.html.  Data  collected  is  put  into  a 
Naval  Postgraduate  School  database  where  it  is  then  analyzed,  and  from 
which  metrics  reports  are  issued. 


79 


%  of  Total  Missions 
%  of  Sqdn  Msns 
MAJCOM 
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Thunderbolt  II 

B-1B 
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B-2 

Spirit 

F-16 

Fighting  Falcon 

U-2 

UAV  (Unmanned 
Aerial  Vehicle) 

Beale  AFB,  CA 

5% 

30% 

ACC 

Creech  AFB,  NV 

0% 

Dyess  AFB,  TX 

17% 

60% 

ACC 

Kunsan  AB,  ROK 

14% 

35% 

PACAF 

Nellis  AFB,  NV 

8% 

10% 

ACC 

0% 

Osan  AB,  ROK 

16% 

50% 

PACAF 

24% 

50% 

PACAF 

Whiteman  AFB,  MO 

16% 

40% 

ACC 

Figure  13.  Summary  of  data  collected  during  data  collection  timeframe,  categorized 
by  air  base  (rows)  and  air  frame  (columns).  Information  on  missions 
describes:  percent  of  total  missions,  percent  of  squadron  missions 
completing  data  collection  forms,  and  major  command  (Air  Combat 
Command  or  Pacific  Air  Forces). 


80 


1 20. 00%  -r 


CloudCeiling  CloudLayers  airfaceVis  VisAloft  In-flight  Icing  IcingRunTinres  Turbulence  Contrails 

Forecasted  Weather  Phenomena 


Figure  14.  Forecast  accuracy  (FAC)  for  forecasts  of  phenomena  (listed  on  horizontal 
axis)  from  which  aviators  inferred  negative  mission  impacts.  Planning 
weather  forecast  FAC  (blue),  mission  execution  forecast  FAC  (purple), 
average  of  PWF  and  MEF  FAC  (yellow).  Note:  large  values  of  FAC  for  a 
given  phenomenon  are  generally  associated  with  a  very  small  sample 
size. 
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Forecasted  Weather  Phenomena 


Figure  15.  Probability  of  detection  (POD)  for  forecasts  of  phenomena  (listed  on 
horizontal  axis)  from  which  aviators  inferred  negative  mission  impacts. 
Planning  weather  forecast  POD  (blue),  mission  execution  forecast  POD 
(purple).  Note:  large  values  of  POD  for  a  given  phenomenon  are 
generally  associated  with  a  very  small  sample  size. 
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120.00% 


Times 

Forecasted  Weather  Phenomena 


Figure  16.  False  alarm  rate  (FAR)  for  forecasts  of  phenomena  (listed  on  horizontal 
axis)  from  which  aviators  inferred  negative  mission  impacts.  Planning 
weather  forecast  FAR  (blue),  mission  execution  forecast  FAR  (purple). 
Note:  large  values  of  FAR  for  a  given  phenomenon  are  generally 
associated  with  a  very  small  sample  size. 
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Not  forecasted  in  PWF  or  MEF  (Total:  17%) 
Forecasted  in  PWF  but  not  in  MEF  (Total:  6%) 

Not  forecasted  in  PWF  but  was  in  MEF  (Total:  6%) 
Forecasted  in  both  PWF  and  MEF  (Total:  30%) 


ceiling  layers  t-storms  rain  sfcvis  vis  aloft  turb  icing  x-wind 


Figure  17.  Percent  of  missions  that  experienced  phenomena  (horizontal  axis)  with 
negative  impacts  that  were/were  not  forecasted  in  the  planning  weather 
forecast  (PWF)  and/or  the  mission  execution  forecast  (MEF). 


Was  Indicated  by 

MEF 


■  Was  Not  Indicated  by 
MEF 


Can't  See  Target  Can't  Takeoff  Can't  Refuel  NoLLTrng  NoA2ATrng  NolnstTmg 


Figure  18.  Percent  of  missions  that  experienced  negative  impacts  (horizontal  axis) 
that  were/were  not  indicated  by  the  mission  execution  forecast  (MEF). 
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Percentage  of  Missions 


Figure  19. 


12 


Based  only  on  PWF 
Based  only  on  MEF 
Based  on  PWF  and  MEF 


Percent  of  missions  for  which  mission  plan  changes  (horizontal  axis)  were 
made  by  aircrews  in  response  to  negatively  impacting  weather 
phenomena  indicated  by  planning  weather  forecast  only  (PWF,  light  blue), 
mission  execution  forecast  only  (MEF,  dark  blue),  or  both  PWF  and  MEF 
(green). 
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Figure  20.  Number  of  missions  for  which  indicated  negative  impacts  (colored 
symbols)  were  inferred  by  aviators  from  planning  weather  forecasts  of 
indicated  weather  phenomena  (horizontal  axis).  Note  that  the  forecasted 
phenomena  associated  with  the  largest  number  of  inferred  negative 
mission  impacts  were  cloud  ceiling  (associated  with  inability  to  see  target), 
cloud  layers  (associated  with  inability  to  see  target),  and  surface  visibility 
(associated  with  inability  to  takeoff  or  land). 
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Figure  21.  Number  of  missions  for  which  indicated  mission  plan  changes  (colored 
symbols)  were  made  based  on  planning  weather  forecasts  of  indicated 
weather  phenomena  (horizontal  axis).  Example:  the  mission  schedule 
was  changed  for  four  missions  based  on  the  PWF  forecast  of  surface 
visibility. 
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Figure  22.  Number  of  missions  for  which  indicated  negative  impacts  (colored 
symbols)  were  inferred  by  aviators  from  mission  execution  forecasts  of 
indicated  weather  phenomena  (horizontal  axis).  Example:  inability  to 
takeoff  or  land  was  inferred  for  14  missions  based  on  forecasts  of  surface 
visibility  in  the  MEFs  for  those  missions. 
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Figure  23.  Number  of  missions  for  which  indicated  mission  plan  changes  (colored 
symbols)  were  made  based  on  mission  execution  forecasts  of  indicated 
weather  phenomena  (horizontal  axis).  Example:  weapons  changes  were 
made  for  four  missions  based  on  the  cloud  layer  forecasts  in  the  MEFs  for 
those  missions. 
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Percentage  of  Missions 
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Figure  24.  Percent  of  missions  deemed  unsuccessful  due  to  negative  impacts  from 
indicated  weather  phenomena  (horizontal  axis).  Seven  percent  of  all 
missions  analyzed  were  deemed  unsuccessful  by  aviators. 
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Figure  25.  Percent  of  missions  for  which  a  positive  mission  contribution  (PMC)  was 
made  by  the  planning  weather  forecast  (PWF,  blue)  or  mission  execution 
forecast  (MEF,  purple).  PMC  criteria  are  summarized  in  blue  text  box 
within  figure  and  are  described  in  detail  in  Chapter  II.  Percentages  based 
in  part  on  number  of  missions  deemed  successful  by  aviators.  For  all 
bases,  10.2%  of  successful  missions  received  a  PMC  from  their  PWFs, 
and  10.2%  of  missions  received  a  PMC  from  their  MEFs. 
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Wiiteman  AFB 


Figure  26.  Percent  of  missions  for  which  a  potential  positive  mission  contribution 
(PPMC)  was  made  by  the  planning  weather  forecast  (PWF,  blue)  or 
mission  execution  forecast  (MEF,  purple).  PPMC  criteria  are  summarized 
in  blue  text  box  within  figure  and  are  described  in  detail  in  Chapter  II. 
Percentages  based  in  part  on  number  of  missions  deemed  unsuccessful 
by  aviators.  For  all  bases,  4.7%  of  missions  received  a  PPMC  from  their 
PWFs,  and  3.7%  of  missions  received  a  PPMC  from  their  MEFs. 
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Generic  Fighter  Aircraft  Data  Collection  Form 


Call  sign:_ 


Date  (mm/dd/yyyy):_ 


_Take-off  time(zulu): _ Land  time  (zulu):_ 


PRE-EXECUTION 

1 .  How  many  hours  prior  to  your  flight  did  you  check  the  weather  provided  by  the  weather 
station  (MEF-mission  execution  forecast)? 

□  6  □  5  □  4  □  3  □  2  □  1  □  0 

2.  What  weather  phenomena  in  the  MEF  were  inferred  to  cause  negative  mission  impacts? 

□  Surface  Visibility  □  Cloud  Ceiling 

□  Clouds  Layers  □  Visibility  Aloft 

□  Other _  □  None 

3.  What  negative  mission  impacts  did  you  infer  from  the  MEF? 

□  Won’t  be  able  to  take-off  □  Won’t  be  able  to  refuel 

□  Won’t  be  able  to  strike  target  □  Won’t  be  able  to  do  low-level  training 

□  Won’t  be  able  to  do  air-to-air  training  □  Won’t  be  able  to  land 

□  Other _  □  None 

4.  What  mission  plan  changes  due  to  the  MEF  were  made  prior  to  your  flight? 

□  Aircrew  change  □  Schedule  (i.e.  time  of  mission) 

□  Too  late  to  make  any  changes  □  Weapon 

□  Type  (e.g.  low-level,  refueling)  □  Other _ 

□  None 

EXECUTION 

5.  What  negative  impacts  due  to  weather  actually  occurred  during  the  mission? 

□  Couldn’t  take-off  □  Couldn’t  refuel 

□  Couldn’t  strike  target  □  Couldn’t  do  low-level  training 

□  Couldn’t  do  air-to-air  training  □  Couldn’t  land 

□  Other _ _  □  None 


6.  Was  the  MEF  accurate? 

□  Yes  □  No,  what  was  inaccurate? 

□  Surface  Visibility 

□  Clouds  Layers 

□  Other _ _ 


□  Cloud  Ceiling 

□  Visibility  Aloft 

□  None 


7.  What  did  you  intend  to  accomplish  on  your  mission? 

□  Low-level  training  □  Air-to-air  combat  training 

□  Strike  target  on  ground  □  Air-to-air  refueling 

□  Instrumentation  training  □  Other _ 


8.  What  did  you  actually  accomplish  on  your  mission? 


□  Low-level  training 

□  Strike  target  on  ground 

□  Instrumentation  training 

□  Nothing 


□  Air-to-air  combat  training 

□  Air-to-air  refueling 

□  Other _ 
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9.  What  was  your  overall  impression  of  the  mission? 

a)  □  Mission  successful 

□  Mission  unsuccessful 

b)  □  No  negative  impacts  forecasted 

□  Negative  impacts  forecasted  inaccurately 

□  Negative  impacts  forecasted  accurately 

c)  □  No  weather  impact  to  mission 

□  Weather  negatively  impacted  mission 

d)  □  Changed  mission  due  to  MEF 

□  No  mission  change  due  to  MEF 

e)  □  Changed  mission  due  to  planning  weather 

□  No  mission  change  due  to  planning  weather 

PLANNING  WEATHER 

10.  How  many  days  prior  to  your  flight  did  you  first  check  the  planning  weather  provided 
by  the  base  weather  station? 

□  6  □  5  □  4  □  3  □  2  □  1  □  0 

11.  Would  you  have  checked  the  weather  earlier  if  it  were  available? 

□  Yes  □  No 

12.  What  mission  plan  changes  due  to  planning  weather  were  made  >12hrs  prior  to  your 
take-off? 

□  Aircrew  change  □  Schedule  (i.e.  time  of  mission) 

□  Weapon  □  Type  (e.g.  low-level,  refueling) 

□  Other _  □  None 


TAWS 

13.  How  are  you  using  Target  Acquisition  Weapon  Software  (TAWS)? 

□  Not  using  (you  are  finished  with  data  collection  form... thanks  for  your  feedback) 

□  Mission  planning 

□  Execution 

□  Both 

14.  Does  TAWS  have  your  weapon  sensor  modeled? 

□  Yes  □  No 

15.  What  are  you  using  TAWS  for? 

□  IR  □  TV  □  Laser  □  NVG  predictions 

16.  Did  you  change  your  weapons  load  or  tactics  based  on  TAWS  predictions? 

□  Yes  □  No 

17.  How  were  the  TAWS  predictions? 

□  Pessimistic  □  Optimistic  □  Accurate 

18.  How  did  you  get  your  weather  for  TAWS? 

□  Download  from  AFWA  □  File  from  the  CWT 
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19.  Was  the  TAWS  weather  accurate? 

□  Yes  □  No 

20.  Would  you  use  TAWS  again? 

□  Yes  □  No 

21.  Rate  the  prediction  software  in  the  areas  below  using  the  following  scale: 
1-not  useful,  2-somewhat  useful  but  not  required,  3-useful,  4-mission  essential 

Pre-mission: 

a.  Route  selection 

b.  Target  viewing  azimuth 

c.  Target  area  tactics 

Inflight: 

a.  Low  level  navigation  12  3  4 

b.  Target  area  tactics  12  3  4 

c.  Target  identification  12  3  4 

d.  Threat  avoidance  12  3  4 

e.  Sensor  cross-check  techniques  12  3  4 


Figure  27.  Proposed  generic  fighter  data  collection  form  for  future  studies. 


12  3  4 
12  3  4 
12  3  4 
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